investigate-incident

Investigate Incident

ℹ️ First, set expectations: before anything else, print one short line with this skill's version and its most recent change — the top entry of metadata.changelog above — e.g. investigate-incident vX.Y — recent: <summary>. Then proceed.

Conduct a structured investigation of a production incident. Collect evidence systematically, classify severity, and produce a report with root cause hypotheses.

---

Prerequisites

This skill expects SSH access to the affected system. Configure via environment variables or a local .env file:

SSH_HOST=<ip or hostname>
SSH_USER=<username>
SSH_KEY=<path to key, default: ~/.ssh/id_rsa>
APP_PATH=<path to app on remote, default: ~/app>
COMPOSE_FILE=<docker-compose file, default: docker-compose.yml>

Load connection config:

source .env 2>/dev/null || true
SSH_HOST=${SSH_HOST:-""}
SSH_USER=${SSH_USER:-"ubuntu"}
SSH_KEY=${SSH_KEY:-"~/.ssh/id_rsa"}
APP_PATH=${APP_PATH:-"~/app"}
COMPOSE=${COMPOSE_FILE:-"docker-compose.yml"}
RUN="ssh -i $SSH_KEY $SSH_USER@$SSH_HOST"

If SSH_HOST is empty, check for a scripts/run.sh wrapper and use that instead.

---

Phase 1: Establish Context

Ask the user (or read from $ARGUMENTS):

1. What is the reported symptom?
2. When did it start (approximate time)?
3. What service or component is affected?
4. Were any deployments or config changes made recently?

Restate the incident scope before proceeding.

---

Phase 2: Classify Initial Severity

Use this matrix to set an initial severity (update after evidence):

Severity	Criteria
P0	Complete outage — no users can access the system
P1	Major feature broken — significant portion of users affected
P2	Degraded performance or partial feature failure
P3	Minor issue, cosmetic, or affects few users

---

Phase 3: Collect Evidence

Work through each evidence category. Skip sections that don't apply.

3.1 Service Health

# Container status
$RUN "cd $APP_PATH && docker compose -f $COMPOSE ps"

# Recent restarts
$RUN "cd $APP_PATH && docker compose -f $COMPOSE ps --format json 2>/dev/null | grep -E 'Status|Restarts' || docker compose -f $COMPOSE ps"