incident-response

Incident Response

Protect users, reduce harm, preserve evidence, and restore a known-safe state during a live incident; production-readiness owns the pre-launch gate and systematic-debugging owns root-cause work once the service is stable.

Workflow

1. Open the incident: Record UTC start, reporter, affected service and environment, candidate artifact or configuration, first symptom, known user impact, and the incident channel; name one commander, one operator, and one communicator even when one person holds all three, and set the next update time.
2. Classify severity: Rank actual and plausible harm across availability, integrity, confidentiality, safety, money, legal exposure, scope, duration, and reversibility; escalate on impact, not noise volume.
3. Keep a timeline: Append observations, hypotheses, decisions, commands, actors, and outcomes with timestamps; keep facts apart from interpretation and preserve original evidence references.
4. Bound the blast radius: Determine affected tenants, regions, versions, data, workflows, dependencies, and time window; check whether it is still expanding and pick the fastest reliable user-impact signal.
5. Generate mitigations: Compare rollback, traffic reduction, feature disablement, isolation, capacity increase, dependency bypass, and safe degraded mode; rank by time to relief, reversibility, secondary risk, and proof quality, and prefer the safest reversible mitigation.
6. Authorize and stabilize: Present the recommended action, exact target, expected signal, abort condition, and rollback before executing; act only within explicit authority, then watch the predeclared health window.
7. Verify recovery: Confirm critical user journeys, error and saturation signals, data correctness, queued work, security posture, and no continued spread; a quiet alert alone is not recovery.
8. Communicate: Send concise updates with impact, current state, actions, next checkpoint, and known unknowns. Do not promise recovery times or send external communications without the responsible owner.
9. Close carefully: End active response only after sustained recovery, cleanup ownership, evidence retention, residual-risk review, and handoff; keep temporary safeguards until their removal has a named test and owner.
10. Run the postmortem: Hold a blameless review with causal analysis, control gaps, concrete owners, and verification dates; keep the narrative inside the evidence and route security design follow-ups to threat-model.
11. Maintain the record: Keep one incident document with:
    • severity, impact, scope, current state, role owners, and next update time;
    • the timestamped timeline and working hypotheses with confirming and disconfirming evidence;
    • each mitigation with its authority, target, abort condition, and result, plus recovery checks and observation window;
    • customer, security, legal, and disclosure coordination gaps, and residual risks with owners and due dates.