incident-postmortem

Incident Response & Postmortem

This skill helps create two key artifacts: postmortem documents (after an incident) and incident runbooks (before incidents happen). Both are critical for building a mature engineering org that learns from failures.

Postmortem Document

The goal of a postmortem is to learn, not to blame. Every production incident is an opportunity to make the system more resilient.

Template

# Incident Postmortem: [Short Title]

**Date of Incident**: [YYYY-MM-DD]
**Duration**: [Start time → End time, with timezone]
**Severity**: [P0/P1/P2/P3]
**Author**: [Name]
**Status**: Draft | Reviewed | Action Items Complete

## Executive Summary
[3-4 sentences max. What happened, who was affected, how long it lasted,
and how it was resolved. A VP should be able to read only this and understand
the incident.]

## Impact
- **Users affected**: [Number or percentage]
- **Revenue impact**: [If measurable]
- **SLA impact**: [Did we breach any SLAs?]
- **Data impact**: [Any data loss or corruption?]

## Timeline
[Chronological sequence of events. Use UTC timestamps.]

| Time (UTC) | Event |
|------------|-------|
| 14:00 | Deploy of v2.3.1 begins |
| 14:05 | Error rate spikes to 15% in monitoring |
| 14:12 | On-call engineer paged |
| 14:15 | War room opened |
| ... | ... |
| 15:30 | Rollback deployed, error rate returns to normal |

## Root Cause
[Technical explanation of what went wrong at the deepest level.
Not "the deploy broke things" but "the migration script assumed
all users had a `preferences` column, but 12% of legacy accounts
created before 2023 didn't have this column, causing NULL reference
exceptions in the UserService.getPreferences() path."]

## Contributing Factors
[What made the incident worse or longer than it needed to be?
Examples: missing monitoring, slow detection, unclear runbooks,
missing feature flags, no canary deployment.]

## Resolution
[What was done to fix the immediate issue. Be specific about
the exact change, not just "we rolled back."]

## Detection
- **How was it detected?** [Alert? Customer report? Someone noticed?]
- **Time to detect**: [Minutes from incident start to first alert/awareness]
- **Could we have detected it sooner?** [What monitoring would have caught it faster?]

## Action Items

| Priority | Action | Owner | Due Date | Status |
|----------|--------|-------|----------|--------|
| P0 | Add NULL check in UserService.getPreferences() | @alice | 2024-01-20 | Done |
| P1 | Add monitoring for migration script failures | @bob | 2024-01-25 | In Progress |
| P2 | Create runbook for database migration rollback | @carol | 2024-02-01 | Not Started |

## Lessons Learned
- **What went well**: [Things that helped during the incident]
- **What went poorly**: [Things that made the incident worse]
- **Where we got lucky**: [Things that could have made it much worse]

## Appendix
[Relevant logs, graphs, metrics screenshots, Slack threads]