incident-runbook-generator

Incident Runbook Generator

Create actionable runbooks for common incidents.

Runbook Template

# Runbook: Database Connection Pool Exhausted

**Severity:** P1 (Critical)
**Estimated Time to Resolve:** 15-30 minutes
**Owner:** Database Team (On-call)

## Symptoms

- Application errors: "connection pool exhausted"
- Increased API latency (>5s)
- Failed health checks
- CloudWatch alarm: `DatabaseConnectionsHigh`

## Detection

- Alert: DatabaseConnectionPoolExhausted
- Metrics: `active_connections > max_connections * 0.9`
- Logs: "Error: connect ETIMEDOUT"

## Immediate Actions (5 min)

1. **Verify the issue**
   ```bash
   # Check current connections
   SELECT count(*) FROM pg_stat_activity;
   ```

1. Identify long-running queries

   SELECT pid, now() - pg_stat_activity.query_start AS duration, query
   FROM pg_stat_activity
   WHERE state = 'active'
   ORDER BY duration DESC
   LIMIT 10;
   

2. Kill blocking queries (if safe)

   SELECT pg_terminate_backend(pid)
   FROM pg_stat_activity
   WHERE state = 'idle in transaction'
   AND now() - state_change > interval '5 minutes';
   

Mitigation (10 min)

1. Scale up connection pool (temporary)

   # Update RDS parameter group
   aws rds modify-db-parameter-group \
     --db-parameter-group-name prod-params \
     --parameters "ParameterName=max_connections,ParameterValue=200"
   

2. Restart application (if needed)

   kubectl rollout restart deployment/api
   

3. Monitor recovery

   watch -n 5 'psql -c "SELECT count(*) FROM pg_stat_activity;"'
   

Root Cause Investigation

Check for:

• Recent deployment (new code with connection leaks)
• Traffic spike (legitimate or DDoS)
• Slow queries holding connections
• Connection pool configuration too small
• Application not releasing connections

Rollback Steps

If caused by deployment:

# Rollback to previous version
kubectl rollout undo deployment/api

# Verify
kubectl rollout status deployment/api