agent-resilience

Agent Resilience

Give an agent's outbound calls three layers of failure handling, applied in the right place. The rule that generalizes: retry only where the call is idempotent, and only where a replica exists to retry against.

When to use

• A slow tool or model call can hang a whole turn instead of failing on a deadline.
• A hard-down dependency is hammered with the full retry budget on every turn.
• The primary model endpoint has no failover, so its outage is a total outage.
• You are about to add a retry to a write — stop and read step 2 first.

Steps

1. Bound every call with a deadline. Pass a context deadline through each tool and model call so a slow dependency returns a clear error instead of hanging the turn.
2. Retry only idempotent reads, never writes. Retrying a non-idempotent action (restart, resolve, charge) can apply it twice. Retry reads with bounded, exponential backoff; leave writes to run exactly once behind human confirmation.
3. Add a circuit breaker for dead dependencies. After N consecutive failures, open the breaker and fail fast (shed load) until a cooldown elapses, then let one trial call test recovery. Keep it opt-in so default behavior stays retry-only, and emit a metric each time it opens.
4. Fail over the model only to a validated fallback. Try a primary model; on a failure before any response, fall back to a secondary you have evaluated. Never switch mid-stream (it splices two answers), and prefer a smaller same-provider model so failover keeps your cost and privacy properties.

Reference implementation

From the AgentOps Open Course, installable with npx skills add MLOps-Courses/agentops-open-course:

• agents/go/resilience/ — context deadlines, bounded retries, and circuit state.
• agents/go/resilience/circuit_test.go — deterministic virtual-time checks with testing/synctest.
• agents/go/model/fallback.go — same-provider fallback before any response is yielded.
• Course chapters 4.5. Guardrails and 5.4. Model Gateway.