Engineering Practice
On-call: multiple-choice review
Crux Multiple-choice synthesis across the on-call unit — symptom vs cause alerting, burn-rate pages, alert fatigue, load caps, runbooks, escalation, and the metrics that keep a rotation honest.
Your altitude — climbing toward senior
ZeroJuniorMiddleSenior
You are at senior altitude — in orbitSix questions that cut across the whole unit. None is a definition to recite — each mirrors a decision a senior makes about what fires a pager at 4 a.m. and what must not.
Goal
Confirm you can connect the unit’s spine: actionable symptom/SLO alerts on the pager, everything else demoted, load capped, runbooks linked, and the rotation measured by % actionable.
Quiz
Completed
A disk-usage threshold pages on-call 40 times a month and self-resolves nearly every time. Why is this more dangerous than having no alert at all?
Heads-up Over-alerting is exactly what makes you miss the real event. A 60–80% false-positive stream desensitizes the responder; coverage breadth is not the same as a trustworthy pager.
Heads-up The cost is human, not storage. Alert fatigue and the desensitization it causes are the failure mode the unit is built around, not log volume.
Heads-up Raising the threshold just delays the same non-actionable cause alert. The question is whether it should page a human at all — page on the symptom (disk full within 1h) and demote the raw threshold.
Quiz
Completed
You must decide what pages a human. Which condition belongs on the pager, and what is the principle?
Heads-up High CPU is frequently fine and self-corrects; it is a cause, not a symptom. Cause alerts fire on non-events and miss real symptoms that don't move that gauge. Page on symptoms; only alert on causes that are definite and imminent.
Heads-up Earliest is not the same as actionable. A near-full pool often recovers; pages should match user-visible harm or SLO risk, not internal mechanics that may never bite.
Heads-up Sensitivity without actionability is how alert fatigue starts — an intermittent tail will fire constantly and self-resolve, training the responder to dismiss it. Page on sustained, budget-threatening symptoms, not individual slow requests.
Quiz
Completed
Google SRE caps a shift at about two incidents and operational work at 50% of an SRE's time. What breaks if you ignore the caps?
Heads-up You can't discipline your way past the math: a third incident means corners get cut, and the corner is the prevention work. The caps protect the engineering half that makes next quarter quieter.
Heads-up Reliability degrades too. Skipped postmortems mean repeat incidents, page volume climbs, responders burn out, and attrition takes operational knowledge out the door — a doom loop, not just lost sleep.
Heads-up They are reliability mechanisms. The 50% ops ceiling exists so the other half goes to engineering that reduces future ops load; ignoring it crowds out exactly that work.
Quiz
Completed
Which metric most directly signals a rotation degrading toward burnout?
Heads-up Rising MTBF means failures are getting rarer — that's good, and it says nothing about how many non-actionable pages are wearing the responder down between failures.
Heads-up Dashboard count is unrelated to pager load or actionability. A team can have excellent dashboards and a brutal, fatiguing pager.
Heads-up Runbooks help MTTR but don't measure whether the rotation's load is sustainable; you can drown in pages while writing excellent runbooks.
Quiz
Completed
A page fires but the primary responder is stuck and hasn't acknowledged. What is the right structural mechanism, and what is the runbook's role here?
Heads-up An unacknowledged page sitting silent is how incidents lengthen. Time-based escalation is precisely the safeguard against a stuck primary; it is not premature noise.
Heads-up Re-paging the same stuck person doesn't add capacity or knowledge. Escalation routes to someone who can actually help; it is a path, not a louder alarm.
Heads-up Even seniors didn't write every service. A runbook turns a 3 a.m. cold start into a procedure and is what lowers MTTR for the median responder; an alert with no runbook isn't ready to page on.
Quiz
Completed
A change is proposed to reduce noise: snooze the noisiest alerts and add Alertmanager group_by, inhibit_rules, and a 30s group_wait. What does this buy you, and what does it not?
Heads-up Grouping reduces redundant pages but a single non-actionable alert is still non-actionable. Fatigue is fixed by deleting or demoting alerts that aren't actionable, not only by de-duplicating them.
Heads-up Snoozing is how coverage quietly dies — the alert is still wired to the pager and will surprise the next responder. Demote it to a ticket or dashboard, or delete it.
Heads-up inhibit_rules mute a warning when a matching critical already fires — useful de-duplication, but it presupposes the alerts were well-designed. It is no substitute for deciding what is actionable upstream.
Recap
The through-line of the unit is one rule applied at every layer: a page must be actionable or it is deleted. Page on symptoms and SLO burn rate, not causes like CPU or disk; demote the rest to tickets and dashboards. Cap load (≈2 incidents/shift, ≤50% ops time) so prevention work survives. Link a runbook to every page, escalate on a timer when a responder is stuck, and use grouping/inhibition to collapse storms — but remember routing only de-duplicates. Measure MTTA, MTTR, and page volume, and steer by the north star: % actionable.