incident-response-and-postmortems

Incident Response and Postmortems

A postmortem's job is to make the system (technical and organizational) less likely to fail the same way twice — not to identify who to blame. Blameless postmortem practice, as documented by Google SRE and by Etsy, exists because blame-seeking postmortems make people hide information, which makes the next incident worse, not better.

Blameless language, concretely

Blameless doesn't mean no accountability — actions still get named owners and deadlines. It means describing what happened in terms of the system and process, not the person:

• Wrong: "Jane forgot to run the migration before deploying."
• Right: "The deploy pipeline did not enforce that the migration step ran before the application deploy step, so the deploy proceeded against a schema it wasn't compatible with."

The second version points at a fixable gap (add a pipeline gate); the first version's only "fix" is "Jane, be more careful," which doesn't survive Jane going on vacation or leaving the team.

When to write one

Trigger a postmortem for: any user-facing outage above your SLO error budget threshold, any data loss or data corruption incident, any security incident, any incident requiring manual/emergency intervention (rollback, hotfix, manual DB repair), or any incident where the on-call was paged outside business hours. Err toward writing one — a short postmortem for a near-miss is cheap; a missed pattern from skipping one is not.

Postmortem structure

1. Summary — one paragraph: what broke, for how long, who/what was affected.
2. Impact — concrete numbers: duration, % of requests/users affected, which hotels/tenants (relevant in a multi-tenant, hotel_id-partitioned system — impact often needs to be scoped per-hotel, not just globally), revenue or SLA impact if known.
3. Timeline — UTC timestamps, one line per event, from first anomaly to full resolution: detection, escalation, diagnosis milestones, mitigation, resolution. Build this from logs/alerts/chat history, not memory — memory reorders events under stress.
4. Root cause — use "5 whys" to go past the first proximate cause. Stopping at "the deploy failed" is rarely the real cause; keep asking why until you hit a systemic gap (missing test, missing alert, missing pipeline gate, unclear ownership).
5. Detection — how was it noticed: alert, customer report, internal discovery? A postmortem where detection was "a customer told us" is itself a finding — that's a monitoring gap.
6. Action items — each one has a named owner and a due date, and is specific enough to be verifiably done or not done. "Communicate better" is not an action item; "add a pipeline gate that blocks deploy until the migration step reports success" is.
7. Lessons learned — what went well (fast detection, good runbook) alongside what didn't, so the good parts get reinforced too.