Postmortems and learning

Why blameless is an engineering choice, not a kindness

Calling a postmortem “blameless” sounds like an HR nicety. It isn’t. It’s the precondition for getting accurate data, and accurate data is the only input from which a real fix can be derived.

The mechanism is simple and brutal: people who fear punishment hide. NIST SP 800-61, the federal incident-handling guide, is built around a final phase — post-incident activity, the “lessons learned” — whose entire value depends on participants telling you what actually happened, in what order, with what they believed at each step. The instant the review can end someone’s quarter, that flow of truth dries up. The on-call engineer stops volunteering “I wasn’t sure the runbook was current, so I improvised,” and you lose the single most load-bearing fact in the whole timeline. You can’t fix a system you can’t see, and fear makes the system invisible.

So blameless is not “we forgive the engineer.” It’s “we assume any competent person, dropped into the exact same context — same tools, same alerts, same time pressure, same stale docs — could have made the same move.” That reframing is the whole trick. It converts the question from who is bad at their job (unanswerable, unfixable, and a lie — the person is usually your best engineer, which is why they were on-call) into what about the environment made the wrong action easy and the right action hard (answerable, and fixable in code, config, and process).

The senior tell is the verb. “The engineer forgot to scope the credentials” stops at a person. “The deploy tool defaulted to broad credentials and offered no warning when a change widened MFA scope” indicts a system you can change tomorrow. Same incident, two different futures.

Root cause is a chain, not a name

The phrase “root cause” misleads juniors into hunting for the one thing. Real incidents are a chain of latent conditions that lined up — what safety engineering calls the Swiss-cheese model: several holes in several layers happened to align, and the hazard passed through all of them at once. Stop at the first hole you find and you patch one slice while the rest stay open.

The discipline is to keep asking “and what allowed that?” until you reach conditions you can actually change — and to branch, because there is rarely a single line. Take the MFA incident:

• MFA got disabled on the admin app. → Why? A config change flipped the enforcement flag.
• Why did one change flip it? The console let a single operator toggle MFA enforcement with no confirmation and no second approver.
• Why did nobody catch it for hours? There was no alert on changes to authentication policy — the SIEM logged it but no rule fired.
• Why was the operator changing SSO config by hand at all? The documented path was broken, so manual console edits were the de facto runbook.

Notice none of the real causes is “the engineer.” They’re a missing confirmation gate, a missing four-eyes control, a missing detection rule, and a broken automated path. Each is a corrective action with an owner. The human is the last hole in the cheese, not the cheese. Map this against MITRE ATT&CK while you’re at it: tagging the incident’s techniques (here, Modify Authentication Process, T1556) tells you which detections you were blind to — and feeds the gap straight into your detection backlog instead of letting it evaporate.

Corrective actions: the only output that counts

A postmortem that ends with “we discussed it and everyone learned a lot” produced nothing. The deliverable is a list of corrective actions, and a corrective action that survives contact with reality has four properties, or it’s theatre:

1. Specific — “harden SSO” is a wish. “Require a second approver and a typed-confirmation step for any change to MFA enforcement policy” is an action.
2. Owned — a named person, not “the team.” Unowned actions are the ones still open at the next incident.
3. Dated — a real due date, tracked in the same system as feature work (Jira, Linear), not buried in a doc nobody reopens.
4. Verifiable — you can prove it shipped. “Added an alert” → there’s now a rule that fires on auth-policy changes, and you tested it with a synthetic change.

The hierarchy matters too. Actions that remove the failure mode beat actions that catch it faster beat actions that remind humans to be careful. “Reminded the team” is the weakest possible control because it relies on memory under pressure — the exact thing that already failed. Prefer eliminate (make the dangerous toggle impossible without approval) over detect (alert when it happens) over educate (tell people not to). Education is the floor, never the fix.

Closing the feedback loop builds resilience

The point of all of this is the loop. NIST’s lessons-learned phase isn’t a closing ceremony — it feeds the next round of preparation. Each incident is a free, expensive lesson about exactly where your system is weak; resilience is the property of an organization that converts those lessons into permanent changes faster than new weaknesses appear.

That loop has teeth only if three things hold. The actions get tracked to done — an action item that’s still “open” at the next quarterly review is a corrective action that didn’t correct anything. The model gets re-run — the gaps you found (the missing alert, the broken automation) flow back into detection rules, threat models, and runbooks, so the system that handles incident N+1 is measurably better than the one that handled N. And the learning gets shared — the postmortem is read beyond the team that wrote it, because the same broken pattern (a privileged toggle with no gate) almost certainly exists in three other consoles you haven’t looked at yet.