design-incident-response

Design Incident Response

Build a structured incident response process that minimizes customer impact, coordinates responders, and prevents recurrence.

Why This Is Best Practice

Adopted by: Google SRE, PagerDuty, Atlassian, AWS, and organizations following NIST SP 800-61 as the federal standard for incident handling Impact: Organizations with a documented incident response process resolve incidents 28% faster and have 50% lower repeat-incident rates than those without (PagerDuty State of Digital Operations, 2022)

Why best: Without a defined process, incidents devolve into heroics — one person firefighting while everyone else watches. A defined incident command structure, clear roles, and a blameless postmortem loop convert incidents into organizational learning.

Steps

1. Define severity levels — Create 3–4 tiers (SEV1–SEV3 or P0–P3) with explicit, measurable criteria: "SEV1 = customer-facing service unavailable for >1% of users or data loss." Avoid vague language like "significant impact."
2. Assign incident command roles — Incident Commander (IC): owns the incident and decisions. Communications Lead: updates stakeholders. Subject Matter Experts (SMEs): do the technical work. No one plays two roles during a SEV1.
3. Create the alert-to-acknowledge SLA — Define time limits per severity: SEV1 = 5 min page, 15 min acknowledge; SEV2 = 30 min. Document escalation path if the primary on-call does not respond.
4. Build the incident channel template — On-call tooling should auto-create a dedicated channel with: severity, incident summary, IC assigned, customer impact, and links to runbooks. Standardize the format so responders orient instantly.
5. Define the update cadence — SEV1: status update every 15 minutes to stakeholders. SEV2: every 30 minutes. Updates follow the format: Current state / Actions taken / Next action / ETA. No open-ended updates.
6. Conduct blameless postmortem within 48 hours — Document timeline, contributing factors (no "root cause" — systems have multiple causes), and action items with owners and due dates. Publish internally.
7. Track postmortem action items — Assign each item to a team with a sprint commitment. Review completion in the next postmortem. Untracked items repeat the incident.