incident-response

Incident Response

Systematic framework for detecting, triaging, mitigating, and resolving incidents.

Severity Levels

Severity	Description	Response Time	Example
SEV1	Service down, data loss, security breach	< 15 minutes	Database unavailable, authentication broken
SEV2	Major degradation, partial outage	< 1 hour	API latency > 5s, 50% of users affected
SEV3	Minor degradation, non-critical	< 4 hours	Single feature broken, workaround available
SEV4	Cosmetic, non-user-facing	< 1 business day	UI typo, non-critical log errors

Incident Commander Checklist

When leading an incident:

• Declare incident — Create incident channel, assign severity, notify stakeholders
• Assess impact — Affected services, user count, business impact
• Form response team — Assign roles (commander, scribe, engineers)
• Establish timeline — Document all actions and observations
• Mitigate — Take immediate actions to restore service
• Communicate — Send status updates every 15-30 minutes
• Resolve — Verify fix, monitor for stability
• Postmortem — Schedule within 48 hours, document learnings

Communication Templates

Initial Incident Alert

🚨 SEV{1-4} Incident: {Service Name} - {Brief Description}

Impact: {User count / % of traffic / Business impact}
Status: Investigating
Timeline:
- {Time} - Incident detected
- {Time} - Response team assembled

Next update in 15 minutes.

Status Update

📊 SEV{1-4} Update: {Service Name}

Status: {Investigating / Mitigating / Resolved}
Impact: {Updated impact assessment}
Actions taken:
- {Action 1}
- {Action 2}
Next update: {Time}

Customer Communication (for SEV1/SEV2)

We're currently experiencing issues with {Service Name} that may affect {specific functionality}.

We're actively working on a fix and will provide updates every {interval}.

Status page: {URL}