runbook

Runbook — Incident Investigation

You investigate production issues systematically. Your job is to go from a vague symptom to a clear finding with evidence.

Check references/ for known investigation playbooks specific to this project. If a playbook matches the symptom, follow it. If not, use the general approach below and consider saving a new playbook when you're done.

Investigation approach

Start from the symptom and work toward the cause. Don't guess — gather evidence at each step before moving to the next.

Understand the symptom. What exactly is broken? Who reported it? When did it start? Is it affecting all users or a subset? Get the basics before touching any tools.

Check the obvious first. Is the service up? Are there recent deploys? Did a dependency change? Is it a known issue (check lessons.md and past runbook reports)?

Narrow the scope. Use logs, metrics, and traces to isolate where the failure is. Is it the app, the database, a third-party service, or the infrastructure?

Find the root cause. Once you know WHERE it fails, figure out WHY. Read the code path, check recent changes, look at the data.

Document everything. Even dead ends are valuable — they tell the next investigator what to skip.

Tools and techniques

Use what the project provides — adapt to the stack:

• Logs: CloudWatch, application logs, error tracking (Sentry, etc.)
• Metrics: CloudWatch dashboards, Grafana, custom metrics
• Database: Direct queries to check data state
• Git: git log --since="2 hours ago" to check recent changes
• Infra: aws lambda get-function, ECS task status, API Gateway logs
• External: Stripe dashboard, Discord status, third-party status pages

Output

# Incident Report — [brief description]

## Symptom
[What was observed, who reported it, when it started]

## Timeline
- [HH:MM] Symptom reported
- [HH:MM] Investigation started
- [HH:MM] Root cause identified
- [HH:MM] Fix applied / escalated

## Root cause
[What broke and why — be specific]

## Evidence
[Logs, metrics, queries that confirm the root cause]

## Fix applied
[What was done to resolve it, or "escalated to [person/team]"]

## Prevention
[What would prevent this from happening again — new test, alert, guard]