agent-incident-response

Agent Incident Response

An operated agent is a running workload with incidents of its own: error-budget burn, latency regression, injection spikes, schema failures, cost blowouts, thumbs-down clusters. Close each one not when service is restored, but when the smallest repeatable check or evidence path would expose it again.

When to use

• An agent workload fires an alert (error budget, p95, cost, guardrail spike, schema failure).
• A quality or cost incident with no alert (thumbs-down cluster, a change that doubled tokens).
• You want every outage to leave behind a test, eval case, alert, or baseline.

The loop: detect → triage → mitigate → review → prevent

1. Detect. Alert on the workload's own health signals (error-budget burn, p95 latency, injection-neutralized spike, structured-output schema failures, missing token telemetry, collector down).
2. Triage a fixed walk. Metric (scope it) → trace (read the failing turn's span tree) → logs (the error text the span summarizes) → audit (what state changed, who approved). They join on the trace id.
3. Mitigate with existing controls. Roll out startup configuration to freeze writes, cap tokens/context, disable a misbehaving opt-in, or select a validated fallback model. Roll committed prompt or code changes back with the prior evaluated image digest. Record the replacement process's readiness time as the first postmortem line.
4. Review blamelessly. Short, factual, about the system: impact, timeline, root cause tied to evidence, "what caught it and what didn't", actions with owners.
5. Prevent — the load-bearing step. Promote the incident to the cheapest reproducible check or evidence path that would catch it again: a deterministic unit/red-team test, an eval case, an alert rule, or a corrected cost measurement. Fix or explicitly approve the cost change before reviewing a new baseline; never normalize unexplained inflation.

Reference implementation

From the AgentOps Open Course (agent modules live under agents/python/src/agent/):