oncall

oncall

Production diagnosis with the same evidence discipline as development: no cause claimed without the log chain and state that prove it, no fix without a covering test.

Load

• ${CLAUDE_PLUGIN_ROOT}/shared/core-contract.md
• ${CLAUDE_PLUGIN_ROOT}/shared/packs/harness-verbs.md
• ${CLAUDE_PLUGIN_ROOT}/shared/packs/evidence.md
• ${CLAUDE_PLUGIN_ROOT}/shared/packs/decision-protocol.md
• ${CLAUDE_PLUGIN_ROOT}/shared/formats/report-style.md
• On hotfix: ${CLAUDE_PLUGIN_ROOT}/shared/packs/release-ops.md

Process

1. Triage: capture the symptom (alert, user report, anomaly), when it started, who is affected. Classify severity: data loss / money / security ⇒ propose rollback or kill-switch FIRST, investigate second; degraded-but-safe ⇒ investigate first.
2. Investigate through observe: first establish expected behavior from the living spec (docs/opc/specs/, when it exists) — regression vs works-as-specified is the first fork. Reconstruct the failing action's causal chain by correlation ID; query state read-only; check traces; diff behavior against the last known-good release — recent release ledger entries, merge commits, and release manifests are the change candidates. Reproduce on local/test with the matching seed when possible.
3. Diagnostic report — docs/opc/incidents/<date>-<slug>.md: timeline, blast radius, root cause with the evidence chain (log lines, queries, commits), contributing factors, and the options considered. No root cause ⇒ say so explicitly and record the leading hypotheses with what would confirm each. Render and lint the same-basename .html. Lead with conclusion, user impact, evidence and next action; explain specialist terms beside their first occurrence.
4. Decide the path (five-piece set to the human when contested):
   • rollback — hand to deploy's rollback path; fastest when the trigger is a recent release;
   • hotfix (expedited path) — minimal fix on a hotfix branch per project flow; classify it by the same engineering triggers as ordinary work. A localized fix stays lite; a complete capability, shared-core refactor, breaking evolution, broad regression surface, or coordinated rollout uses build. Preserve RED/GREEN evidence and targeted regression, then ship → deploy with the incident referenced. Expedited ≠ unverified;
   • mitigation — config/flag/scale change with its own evidence and a revert note.
5. Always, after stabilization:
   • append the root cause to docs/opc/error-ledger.jsonl (tag it — incidents feed retro's recurrence detection like any other failure);
   • for P0/P1, irreversible-risk, false-green, or recurring incidents, link a project benchmark case proving GREEN → RED → GREEN, or record a human-approved waiver;
   • write the long-term fix proposal: what structural change prevents the class, sized as a lite task, one build increment, or an explicit brainstorm/architect decision task; link it in the report;
   • record any harness gap the investigation exposed (missing correlation ID, missing dump, missing prod-safe spec) — these become harness backlog.