Incident Metrics That Matter

What Makes a Metric Worth Tracking?

Not every number that can be measured deserves a dashboard. Good incident metrics share three characteristics: they’re actionable, they reveal trends over time, and they connect directly to business outcomes.

When your MTTR increases, you know something changed in your response process. When MTTA stays consistently low, you know your alerting works. When MTBF trends upward, you know your systems are getting more reliable.

Bad metrics just generate reports nobody reads.

Mean Time to Detect (MTTD)

What it measures: How long issues exist before your team knows about them.

You cannot fix what you do not see. MTTD captures the delay between when something breaks and when your monitoring notices. Low MTTD means your monitoring catches problems quickly. High MTTD means users report issues before your systems do.

How to calculate it:

MTTD = Total detection time / Number of incidents

Detection time starts when the issue occurs and ends when your monitoring triggers an alert. If your API starts returning errors at 2:00 PM but monitoring doesn’t alert until 2:15 PM, your MTTD for that incident is 15 minutes.

What good looks like:

• Critical services: Under 5 minutes
• Important services: Under 15 minutes
• Non-critical services: Under 30 minutes

How to improve it:

Monitor what matters. Track error rates, response times, and business transactions—not just infrastructure metrics. Multi-region monitoring catches geographic issues that single-region checks miss. Smart alerting thresholds reduce false positives while catching real problems early.

Mean Time to Acknowledge (MTTA)

What it measures: How quickly someone responds to alerts after they fire.

Your monitoring detected the problem. Now someone needs to look at it. MTTA measures the gap between alert notification and human acknowledgment.

Long MTTA indicates problems: alert fatigue, notification routing issues, or unclear on-call schedules. When engineers ignore alerts because most are false positives, MTTA suffers.

How to calculate it:

MTTA = Total acknowledgment time / Number of incidents

Acknowledgment time starts when the alert fires and ends when a responder acknowledges it. If an alert triggers at 3:00 AM but the on-call engineer acknowledges it at 3:08 AM, MTTA for that incident is 8 minutes.

What good looks like:

• Critical incidents: Under 5 minutes
• High-priority incidents: Under 10 minutes
• Medium-priority incidents: Under 20 minutes

How to improve it:

Fix your alerting. Reduce false positives so engineers trust notifications. Verify on-call schedules route alerts to available people. Test notification channels to ensure alerts reach responders. Use escalation policies so backup engineers get notified if primary responders miss alerts.

Mean Time to Resolve (MTTR)

What it measures: How long incidents take from detection to full resolution.

MTTR is the most widely tracked incident metric because it directly measures customer impact. The faster you resolve incidents, the less downtime users experience.

How to calculate it:

MTTR = Total resolution time / Number of incidents

Resolution time starts when monitoring detects the issue and ends when service is fully restored and verified. If an incident starts at 10:00 AM and service is restored at 10:45 AM, MTTR is 45 minutes.

What good looks like:

This varies dramatically by industry and team size. Track your own baseline, then improve it:

• Month 1: Establish baseline (e.g., 90 minutes average)
• Month 3: 20% improvement (72 minutes)
• Month 6: 40% improvement (54 minutes)

How to improve it:

Structure your response process. Maintain runbooks for common incidents. Track who’s working on what during active incidents. Document resolution steps for post-mortems. Automate repetitive tasks like rollbacks or service restarts.

The highest-impact improvements often come from coordination, not technical speed. Eliminating confusion about who should investigate saves more time than faster troubleshooting.

Mean Time Between Failures (MTBF)

What it measures: How often systems fail.

While detection, acknowledgment, and resolution metrics track response effectiveness, MTBF tracks system reliability. Long MTBF means your infrastructure stays stable. Short MTBF means recurring problems.

How to calculate it:

MTBF = Total operational time / Number of failures

If your service runs for 720 hours in a month and experiences 6 incidents, MTBF is 120 hours (5 days).

What good looks like:

• Production systems: Weeks to months between incidents
• Experimental features: Days to weeks between incidents
• Legacy systems: Track trend toward improvement

How to improve it:

Fix root causes, not symptoms. Conduct thorough post-mortems after incidents. Identify patterns in failures—the same database query timing out, repeated deployment issues, recurring third-party service problems.

MTBF improves when you stop fighting the same fire repeatedly and start preventing ignition.

Metrics That Distract More Than They Help

Some commonly tracked metrics sound useful but rarely drive improvement:

Incident Count: High incident counts don’t necessarily indicate problems if you’re detecting and resolving issues quickly. Low counts might mean good reliability—or blind spots in monitoring.

Number of Alerts Fired: This measures alerting volume, not effectiveness. Ten accurate alerts are more valuable than one hundred false positives.

Time Spent in Each Status: Tracking how long incidents spend in “Investigating” versus “Resolving” adds complexity without clear improvement paths. Focus on total resolution time instead.

Perfect Resolution Rate: Striving for 100% first-time resolution sounds good but often leads to prolonged incidents as teams pursue perfect fixes under pressure. Restore service quickly, then implement lasting fixes during maintenance windows.

Context Matters More Than Numbers

A metric without context is just a number. MTTR of 30 minutes means different things for different incident types.

Severity matters: Critical incidents affecting all users should resolve faster than minor issues impacting a single customer. Track MTTR by severity level, not as a single global number.

Type matters: Database failures, network outages, and application bugs have different resolution characteristics. What’s normal for infrastructure incidents might be terrible for configuration problems.

Trends matter most: One incident with 3-hour MTTR doesn’t mean your process is broken. Consistent increase in MTTR over three months signals problems. Look for patterns, not anomalies.

How to Actually Use These Metrics

Tracking numbers accomplishes nothing. Using them to improve response processes creates value.

Weekly reviews: Look at last week’s incidents. Which had high MTTR? Why? Were runbooks followed? Did the right people respond?

Monthly trends: Compare this month to last month. Is MTTA increasing? That’s a signal alert fatigue is growing. Is MTBF decreasing? Time for deeper root cause analysis.

Quarterly goals: Set specific, measurable targets. “Reduce P1 MTTR from 45 minutes to 30 minutes by end of quarter.” Then track progress weekly.

Post-incident analysis: After every major incident, examine all four metrics. How quickly did you detect? How fast did someone acknowledge? How long to resolve? When was the last similar failure?

Building a Metrics Culture

Metrics reveal problems. Humans fix them. The wrong culture makes metrics counterproductive.

Never punish people based on metrics. If engineers fear looking bad, they’ll game the metrics. Late acknowledgment? Mark it as hardware delay. Long resolution time? Classify it as lower severity. Metrics lose meaning when teams have incentives to manipulate them.

Use metrics for process improvement, not performance reviews. When MTTR increases, ask what changed in the process, not who performed poorly. System problems require system fixes.

Share metrics widely. Teams that see their metrics improve feel ownership over incident response quality. Hide metrics in management reports and engineers disengage from improvement efforts.

Celebrate improvement, not perfection. Reducing average MTTR from 60 minutes to 45 minutes deserves recognition. Obsessing over hitting exactly 30 minutes creates stress without proportional value.

Tracking Metrics in Practice

Manual metric tracking fails because it’s tedious. Engineers resolving incidents at 3 AM will not remember to log detection time versus resolution time.

Modern incident management platforms track these metrics automatically. Platforms like Upstat record incident timelines automatically—when incidents start, when responders acknowledge, when status changes, and when resolution completes. This provides accurate duration tracking and historical trends without manual data entry.

Built-in analytics show MTTR trends over time, acknowledgment patterns by team, and incident frequency by service. Teams can identify improvement opportunities through actual data instead of intuition.

Starting Your Metrics Practice

If you’re not tracking incident metrics today, start small:

Week 1: Track MTTR manually for one week. When incidents occur, record start time and resolution time. Calculate average at week’s end.

Week 2: Add MTTA. Note when alerts fire and when someone acknowledges them.

Week 3: Establish baseline. Now you know where you stand. Pick one metric to improve.

Month 2: Add automated tracking. Manual tracking provides proof of concept. Automation makes it sustainable.

Month 3: Review trends. Are metrics improving? What changed? What didn’t work?

The Metrics That Actually Matter

Stop tracking vanity metrics. Stop creating dashboards nobody reads. Start tracking four numbers that directly connect to incident response quality: MTTD reveals monitoring effectiveness, MTTA shows alerting health, MTTR measures response efficiency, and MTBF indicates system reliability.

Track them consistently. Review them regularly. Use them to drive specific improvements. Celebrate progress.

Good metrics won’t fix incident response by themselves. But they’ll show you exactly where your process breaks down—and prove when your improvements actually work.