Why On-Call Performance Reviews Matter
On-call engineers carry significant operational responsibility. When systems fail at 2 AM, these individuals respond under pressure, often with incomplete information, making quick decisions that affect customer experience and business outcomes.
Yet many organizations either avoid evaluating on-call performance entirely or measure it using metrics that create perverse incentives. Counting resolved incidents without considering alert quality rewards quantity over quality. Tracking response time without acknowledging shift difficulty penalizes engineers who inherit complex problems.
Effective on-call performance reviews balance quantitative metrics with qualitative context. They distinguish between individual performance and system performance. They acknowledge that a high-performing engineer working with poor tooling will struggle, while an average engineer with excellent runbooks and automation can excel.
Done correctly, performance reviews improve both operational reliability and engineer well-being by identifying what works, what doesn’t, and what needs improvement.
Core Metrics for On-Call Evaluation
Response Time Metrics
Mean Time to Acknowledge (MTTA) measures how quickly engineers respond to alerts. Industry benchmarks suggest under 5 minutes for critical alerts during scheduled shifts. However, raw numbers miss important context.
An engineer who consistently acknowledges alerts within 2 minutes demonstrates strong responsiveness. But if their shifts coincide with business hours when incident volume is naturally lower, that metric alone doesn’t tell the complete story. Conversely, an engineer averaging 4-minute acknowledgment during high-volume overnight shifts may actually be performing better relative to difficulty.
Track MTTA alongside shift characteristics: time of day, incident volume, alert severity distribution. Normalize metrics across different shift patterns to enable fair comparisons.
Resolution Effectiveness
Mean Time to Resolution (MTTR) tracks how long incidents remain open from alert to closure. This metric directly impacts customer experience and business operations, making it central to on-call effectiveness.
Yet MTTR also requires context. An engineer who resolves ten minor database connection issues quickly demonstrates different capabilities than one who methodically diagnoses and resolves a single complex distributed system failure. Both provide value. Both deserve recognition.
Segment MTTR by incident severity and type. Track escalation rates—how often engineers recognize problems beyond their expertise and appropriately escalate rather than struggling alone. Measure quality of resolution: what percentage of incidents recur within 24 hours indicating incomplete fixes.
Alert Quality Interaction
False positive rates reveal both alert configuration problems and engineer judgment. An engineer who frequently dismisses or ignores alerts that turn out to be legitimate issues needs coaching. An engineer who diligently responds to alerts that consistently prove non-actionable is suffering from system problems, not performance problems.
Track the ratio of acknowledged alerts that require actual action versus those that resolve themselves or prove false positives. When engineers report alert fatigue, examine their specific alert patterns. Individual performance reviews should credit engineers who identify and document problematic alerts even when they can’t fix the underlying monitoring configuration themselves.
Incident Documentation Quality
Post-incident documentation provides lasting value beyond immediate resolution. Well-documented incidents create knowledge for future responders, support postmortem analysis, and enable systemic improvements.
Evaluate documentation completeness: Does the incident record contain sufficient detail for someone unfamiliar with the problem to understand what happened? Does it capture troubleshooting steps attempted? Does it identify contributing factors and
follow-up work?
Quality matters more than length. A concise incident summary with clear timeline, actions taken, and resolution path proves more valuable than verbose narratives that obscure key information.
Qualitative Evaluation Factors
Communication During Incidents
Technical skills resolve incidents, but communication skills determine whether resolution happens efficiently or chaotically. Evaluate how engineers coordinate during major incidents.
Do they provide clear status updates to stakeholders? Do they effectively communicate with other responders? Do they escalate appropriately when problems exceed their expertise? Do they update incident channels with current understanding and next steps?
Peer feedback proves particularly valuable for assessing communication effectiveness. Engineers working together during complex incidents can evaluate collaboration quality better than managers reviewing incident logs after the fact.
Learning and Growth
Strong on-call performance includes continuous improvement. Evaluate whether engineers demonstrate learning from incidents.
Do they update runbooks after encounters with undocumented scenarios? Do they propose automation for repetitive manual tasks? Do they share knowledge with team members about tricky problems they’ve solved? Do they identify monitoring gaps or alert quality issues?
Track runbook contributions, postmortem participation, and proactive improvement proposals as indicators of growth mindset and operational maturity.
Collaboration and Support
On-call doesn’t mean working alone. Evaluate how engineers support teammates and seek support themselves.
Do they assist other on-call engineers when needed? Do they help onboard new team members to on-call responsibilities? Do they participate in postmortems constructively? Do they escalate appropriately rather than struggling in isolation?
Collaborative behavior creates more resilient on-call systems by distributing knowledge and building team capacity.
Framework for Fair Evaluation
Distinguish System Performance from Individual Performance
Many on-call metrics measure system health more than individual effectiveness. High incident volume might indicate fragile infrastructure, not poor engineer performance. Long resolution times might reflect missing automation, inadequate tooling, or complex technical debt.
Separate evaluation discussions into two tracks: What can this engineer improve individually, and what systemic issues constrain their effectiveness? Both conversations matter, but conflating them creates unfair assessments and misses opportunities for organizational improvement.
Account for Shift Difficulty Variation
Not all on-call shifts carry equal challenge. Overnight shifts disrupt sleep and reduce cognitive performance. Weekend shifts interfere with personal time differently than weekday shifts. High-volume incident periods create more stress than quiet weeks.
When comparing performance across engineers, normalize for shift characteristics. Absolute metrics favor engineers with easier shifts. Relative metrics compared to historical averages for similar shift patterns provide fairer comparison.
Balance Individual Metrics with Team Outcomes
Individual performance evaluation shouldn’t create incentives that harm team outcomes. Rewarding individual incident resolution counts might discourage knowledge sharing or escalation. Penalizing slow MTTR might encourage quick superficial fixes over thorough root cause analysis.
Evaluate both individual contributions and team behaviors that support collective success. Credit engineers who help teammates, document knowledge, and improve systems even when those activities don’t directly appear in their personal incident metrics.
Incorporate Self-Assessment
Engineers understand context that metrics miss. Invite self-assessment as part of performance reviews. What went well during their on-call periods? What felt challenging? What would have helped them be more effective?
Self-assessment provides valuable signal about engineer confidence, perceived support, and areas where they feel they need development. It also surfaces systemic issues that might not be visible from metrics alone.
Conducting Performance Conversations
Focus on Growth, Not Blame
Frame performance reviews as development conversations, not judgment sessions. The goal is improving operational effectiveness and individual capabilities, not assigning blame for incidents or system failures.
Ask “What would have made that incident easier to handle?” rather than “Why did that incident take so long?” The first question invites problem-solving. The second invites defensiveness.
Acknowledge that incidents represent system failures, not personal failures. Engineers who respond to incidents are addressing problems they didn’t create, often under pressure and time constraints.
Recognize Outstanding Performance
On-call responsibility carries real cost. Engineers sacrifice sleep, personal time, and cognitive bandwidth for organizational operational needs. Recognize exceptional performance explicitly during reviews.
Highlight specific incidents where engineers demonstrated strong problem-solving, effective communication, or excellent judgment. Acknowledge consistent reliability and responsiveness. Credit proactive improvements to monitoring, automation, or documentation.
Recognition matters particularly for on-call work because much of it happens outside regular business hours when visibility to leadership is limited.
Create Actionable Development Plans
Effective performance reviews identify specific, achievable improvements. Vague feedback like “improve incident response” provides no actionable guidance. Specific feedback like “focus on documenting troubleshooting steps in incident timelines” enables concrete action.
Partner with engineers to create development plans that address identified gaps while providing necessary support. If an engineer needs faster MTTR for database incidents, provide training on database troubleshooting, pair them with experienced engineers during on-call shifts, or invest in better database diagnostic tooling.
Development plans should balance what engineers need to learn with what systems need to improve. Both dimensions contribute to better operational outcomes.
Common Pitfalls to Avoid
Comparing Across Unequal Shifts
Ranking engineers by raw metrics without accounting for shift difficulty creates unfair comparisons. An engineer with daytime business-hours shifts handling low-severity alerts will naturally show better response times than an engineer covering overnight shifts during major incident patterns.
Context matters. Evaluate engineers relative to their specific shift characteristics rather than against organization-wide averages that obscure important differences.
Ignoring Alert Quality Problems
Low MTTA combined with high false positive rates indicates alert configuration problems, not poor engineer performance. When engineers respond promptly but repeatedly encounter non-actionable alerts, fix the alerts, don’t penalize the engineers.
Similarly, high MTTR during shifts with particularly complex or ambiguous alerts reflects system gaps in troubleshooting tools, documentation, or architecture clarity. Address those gaps systemically.
Neglecting Non-Response Contributions
On-call performance reviews naturally focus on incident response metrics because those are quantifiable. But on-call engineers also contribute through runbook creation, postmortem participation, alert quality improvement, and teammate support.
These contributions matter for long-term operational health even though they don’t appear in response time dashboards. Incorporate them explicitly in evaluation frameworks.
Building Sustainable On-Call Culture
Effective performance evaluation supports sustainable on-call culture. When reviews focus purely on metrics without acknowledging context, engineers feel surveillance rather than support. When reviews ignore poor alert quality or inadequate tooling, engineers feel blamed for systemic problems.
Balanced evaluation recognizes both what individuals control and what systems determine. It celebrates strong performance while identifying improvement opportunities. It distinguishes learning moments from accountability failures.
Most importantly, it treats on-call responsibility as shared organizational commitment requiring appropriate support, tooling, and recognition rather than burden carried by engineers alone.
Platforms like Upstat provide comprehensive on-call analytics including incident response metrics, alert acknowledgment patterns, and resolution time tracking across team members. When combined with qualitative evaluation and contextual understanding, these metrics enable fair performance assessment that improves both operational effectiveness and engineer experience. Teams can track MTTR trends, identify alert quality issues, and measure response patterns while maintaining the context necessary for meaningful performance conversations.
Performance reviews done well make on-call better for everyone. Engineers receive recognition and development support. Organizations gain operational insight. Systems improve through identified gaps. That balance makes the difference between on-call as burden and on-call as sustainable responsibility.
Explore In Upstat
Track on-call metrics like incident response times, alert acknowledgment patterns, and resolution effectiveness with comprehensive reporting and team analytics.