incident-management

Incident management

An incident is any unplanned disruption significant enough that normal work stops until it is resolved. The discipline exists because the instincts that serve people well in ordinary work — investigate thoroughly, decide carefully, keep everyone informed — all fail under time pressure unless someone has structured them in advance.

This covers operational incidents generally. For security incidents, where evidence preservation and disclosure obligations change the order of operations, see security:incident-response.

Declare it, and say so out loud

The most expensive minutes are the ones spent deciding whether this is an incident. Set a low threshold for declaring and accept that some declarations will be withdrawn — an incident stood down after twenty minutes costs far less than one that ran for two hours as a conversation between three people who each assumed someone else had it.

Severity should be defined in advance, in terms of customer impact rather than internal inconvenience, with each level carrying a stated response: who is notified, how fast, and who can be woken.

Name a commander who does not fix anything

One person owns the incident: they decide, they sequence, they assign. They should not be the person with their hands in the system — the moment the commander starts debugging, nobody is running the incident and the timeline stops being kept.

Separate three roles even in a small response: the commander, the people restoring service, and one person handling communication. Combining the first and third is survivable; combining the first and second is how incidents run long without anyone noticing they have.

Restore first, understand later

The goal during the incident is service restored, not cause understood. Roll back, fail over, disable the feature, add capacity — whatever returns the customer to working. Diagnosis is tomorrow's work, and pursuing it while people are affected is the most common way a thirty-minute outage becomes a four-hour one.