incident-response

Incident Response

Core Principle

Incidents are inevitable. The goal is not to prevent all incidents but to detect them fast, resolve them fast, and learn from them. Blamelessness is non-negotiable -- people make better decisions when they are not afraid of punishment.

---

1. Incident Severity Levels

Severity	Impact	Response Time	Examples
SEV1 -- Critical	Complete service outage affecting all users. Revenue loss active. Data integrity risk.	Respond within 15 minutes. All-hands incident response.	Site is down. Payment processing is broken. Data breach detected.
SEV2 -- Major	Significant degradation affecting many users. Major feature is broken.	Respond within 30 minutes. Dedicated incident commander + on-call.	Checkout flow broken for 50%+ users. Login failures spiking. API latency > 10x normal.
SEV3 -- Minor	Partial degradation affecting some users. Workaround exists.	Respond within 2 hours. On-call engineer handles.	One region degraded. Non-critical feature broken. Elevated error rate for one endpoint.
SEV4 -- Low	Cosmetic issue or minor bug with minimal user impact.	Respond next business day. Tracked as a bug ticket.	UI rendering glitch. Incorrect error message. Slow non-critical background job.

Escalation Rules

• Any on-call engineer can declare a SEV1 or SEV2. Do not wait for manager approval.
• SEV1 automatically pages the engineering manager and VP of Engineering.
• SEV2 automatically pages the team lead.
• If an incident is not resolved within 1 hour, escalate severity by one level.
• When in doubt, escalate. It is always better to over-communicate.

---

2. Blameless Post-Mortem Template

Write a post-mortem for every SEV1 and SEV2 incident within 72 hours. SEV3 incidents get a post-mortem if the team decides it warrants one.

# Post-Mortem: [Incident Title]

**Date:** [YYYY-MM-DD]
**Severity:** [SEV1 / SEV2 / SEV3]
**Duration:** [Start time - End time, total duration]
**Author:** [Name]
**Reviewers:** [Names of people who reviewed this document]

---

## Summary

[2-3 sentences describing what happened, what the user impact was, and how it was resolved.
Example: "On 2025-06-15, the checkout service became unavailable for 47 minutes due to a
database connection pool exhaustion caused by a missing index on the orders table. Approximately
12,000 users were unable to complete purchases, resulting in an estimated $85,000 in lost revenue.
The issue was resolved by adding the missing index and increasing the connection pool size."]

---

## Impact

- **Users affected:** [Number or percentage]
- **Duration of impact:** [Minutes/hours]
- **Revenue impact:** [Estimated $, if applicable]
- **Data impact:** [Any data loss or corruption? If so, describe.]
- **SLO impact:** [How much error budget was consumed?]

---

## Timeline (all times in UTC)

| Time | Event |
|---|---|
| 14:00 | Deployment of v2.3.1 to production |
| 14:12 | Monitoring alert fires: "High error rate on /api/checkout" |
| 14:15 | On-call engineer acknowledges alert |
| 14:18 | Incident declared as SEV2, Slack channel created |
| 14:25 | Root cause identified: database connection pool exhaustion |
| 14:30 | Mitigation applied: increased connection pool from 20 to 50 |
| 14:35 | Error rate returns to normal |
| 14:45 | Incident resolved, monitoring confirms recovery |
| 14:50 | Severity downgraded, incident closed |

---

## Root Cause

[Detailed technical explanation of what caused the incident. Be specific.
Example: "The v2.3.1 deployment added a new query to the checkout flow that joins the orders
and order_items tables. This query lacks an index on order_items.order_id, causing a full
table scan on every checkout request. Under production load (500 req/s), this query consumed
all 20 database connections within 12 minutes, causing subsequent requests to fail with
connection timeout errors."]

---

## Contributing Factors

[Factors that made the incident more likely or made it harder to detect/resolve. These are NOT
the root cause but contributed to the severity or duration.]

- [ ] The query was not load-tested before deployment.
- [ ] The connection pool size (20) was too small for the traffic volume.
- [ ] The staging environment has 1/100th the data volume, so the missing index was not noticeable.
- [ ] The deployment happened on a Friday afternoon with reduced staffing.

---

## Detection

- **How was the incident detected?** [Monitoring alert / Customer report / Engineer noticed]
- **Time to detect:** [Minutes from start of impact to first alert]
- **Was the alert actionable?** [Yes/No -- did the alert message point to the problem?]
- **What would have detected this faster?** [e.g., "A query latency alert on p99 > 1s would have
  fired 5 minutes earlier than the error rate alert."]

---

## Resolution

[What was done to resolve the incident? Be specific about the steps taken.]

1. Increased database connection pool from 20 to 50 (immediate mitigation).
2. Added index on `order_items.order_id` (root cause fix).
3. Deployed v2.3.2 with the index migration.

---

## Action Items

| Action | Owner | Priority | Due Date | Status |
|---|---|---|---|---|
| Add index on order_items.order_id | @alice | P0 | 2025-06-16 | Done |
| Add query latency alert (p99 > 1s) | @bob | P1 | 2025-06-20 | In Progress |
| Add slow query logging (> 500ms) | @alice | P1 | 2025-06-20 | To Do |
| Add load test for checkout flow with production data volume | @charlie | P2 | 2025-06-30 | To Do |
| Review connection pool sizing for all services | @bob | P2 | 2025-07-01 | To Do |
| No Friday deployments policy for critical services | @team-lead | P2 | 2025-06-20 | To Do |

---

## Lessons Learned

**What went well:**
- Alert fired within 12 minutes of the issue starting.
- On-call engineer responded within 3 minutes of the alert.
- Root cause was identified quickly (10 minutes).

**What could be improved:**
- Staging environment data volume should match production for performance testing.
- Connection pool sizes should be reviewed as traffic grows.
- Deploy-time checks should verify that new queries have appropriate indexes.

---

## Appendix

[Include relevant graphs, logs, or links to dashboards that illustrate the incident.]