incident-response

Manage declared incidents end-to-end

Mitigate first, understand second, learn always. Structure follows the Google SRE book ("Managing Incidents", "Postmortem Culture") and, for security incidents, NIST SP 800-61 Rev. 3 (detect → respond → recover, feeding back into preparation). In Gaffer you usually act on artefacts — classification notes, the incident timeline, the postmortem, the follow-up tickets — not on production: you cannot deploy, roll back or reach cloud consoles, and never run a production-changing command. Mitigations are recommendations a human executes; if one is needed now, raise request_decision with severity: human_required (or security_required for a security incident).

Severity and declaration

Declare an incident (SRE book criteria) if any is true: a second team is needed; the problem is user-visible; it is unresolved after an hour of focused work. When in doubt, declare at the higher severity and downgrade later. Any loss of acknowledged data is SEV1 regardless of how few users saw it.

Procedure

1. Confirm it is real. Compare the firing signal to its baseline; check whether the alert rule, the deploy, or the telemetry pipeline changed recently. A false positive ends here: record why, and file an alert-tuning follow-up. Do not silence without a record.
2. Classify and assign roles. Severity from the table. Roles: Incident Commander (coordinates, decides, owns the state document), Operations lead (the only one changing systems), Communications lead (stakeholder updates on a fixed cadence), Planning lead (tracks follow-ups, handoffs). One person may hold several roles for SEV3/4; the IC role is always explicit and handed off explicitly.
3. Open a live state document and keep a UTC timeline: every observation, decision, and action with who/when. It becomes the postmortem's timeline.
4. Mitigate before root cause. Fastest safe action that stops user harm: roll back the last change, disable the feature flag, shift traffic, shed load, stop the writer that is corrupting data. For suspected data loss, stop further writes to the affected store before anything else. For security incidents, contain (revoke credentials, isolate hosts) while preserving evidence — snapshot logs and disks before restarting or deleting anything. Write the chosen action into the state document for a human to execute; you do not execute it.
5. Diagnose from the first anomaly forward. Which golden signal moved first, in which component, right after which change? Check deploys, config changes, dependency status, traffic shape. For data-integrity incidents look specifically for concurrent writers, read-modify-write without locking or versioning, temp-file name collisions, and time-based locks or leases that another process could break while the holder was paused (GC, SIGSTOP, VM migration) — the class of defect that shipped twice in live runs.
6. Recover and verify. The code fix goes on this ticket's branch; a human deploys it. Write the recovery check into the state document: the SLI back within SLO over a full short window and, for data incidents, what was acknowledged reconciled against what was persisted. State the impact numerically from the data you were given.
7. Postmortem — mandatory for SEV1/SEV2, any data loss, any on-call intervention (rollback, traffic reroute), any incident found by a human rather than monitoring. Draft within 2 business days. Sections: summary; impact (users, duration, error budget consumed, data affected); UTC timeline; root cause(s) and trigger; contributing factors; detection (how, how fast, what would have caught it sooner); what went well; where we got lucky; action items.
8. Action items each have an owner, a priority, a due date, and a tracking reference; at least one per contributing factor; prefer items that prevent a class of failure (a test schedule that pauses the lock holder, an alert on lease takeovers) over "be more careful". Evidence the postmortem with the record-evidence skill, then stop.