incident-response

Incident Response

Purpose

Restore service quickly, then learn from what happened. The two mistakes that define bad incident response are debugging before mitigating, and a postmortem whose action items are never done.

When to Use

• An active production incident.
• Writing a postmortem.
• Building runbooks and an on-call practice.
• Reviewing a pattern of repeated incidents.

Capabilities

• Triage: severity assessment and impact scoping.
• Mitigation: rollback, feature flag, traffic shift, scaling.
• Incident command: roles, communication, and timeline keeping.
• Blameless postmortems and contributing-factor analysis.
• Runbook authoring.

Inputs

• The alert, the symptom, and the user impact.
• Recent changes: deploys, config changes, flag flips, dependency incidents.
• The system's rollback and degradation options.

Outputs

• Service restored, with the mitigation recorded.
• A timeline of what happened and what was done.
• A postmortem with owned, dated action items.

Workflow

1. Establish impact — Who is affected, how badly, and is it growing? This determines severity and whether to wake more people.
2. Mitigate before diagnosing — If a deploy went out in the last hour, roll it back. Understanding why it broke can wait; users cannot. This is the single most important rule and the one most often violated by engineers who want to know the answer.
3. Assign roles — For anything beyond a small incident: an incident commander (decides), a communications lead (updates stakeholders), and operators (execute). The commander does not type.
4. Communicate on a cadence — Status updates at fixed intervals, even when the update is "still investigating". Silence is interpreted as chaos.
5. Record the timeline as you go — Timestamped actions and observations, in the incident channel. Reconstructing it afterwards produces a fiction.
6. Postmortem within a week — Blameless, focused on contributing factors, with action items that have an owner and a date.