agent-resilience

Agent Resilience

Give an agent's outbound calls three layers of failure handling, applied in the right place. The rule that generalizes: retry only where the call is idempotent, and only where a replica exists to retry against.

When to use

• A slow tool or model call can hang a whole turn instead of failing on a deadline.
• A hard-down dependency is hammered with the full retry budget on every turn.
• The primary model endpoint has no failover, so its outage is a total outage.
• You are about to add a retry to a write — stop and read step 2 first.

Steps

1. Bound every call with a deadline. Wrap each tool/model call so exceeding a timeout raises a clear error instead of hanging. Run sync tools in a worker thread so the deadline can actually fire on the event loop.
2. Retry only idempotent reads, never writes. Retrying a non-idempotent action (restart, resolve, charge) can apply it twice. Retry reads with bounded, exponential backoff; leave writes to run exactly once behind human confirmation.
3. Add a circuit breaker for dead dependencies. After N consecutive failures, open the breaker and fail fast (shed load) until a cooldown elapses, then let one trial call test recovery. Keep it opt-in so default behavior stays retry-only, and emit a metric each time it opens.
4. Fail over the model only to a validated fallback. Try a primary model; on a retriable 429/5xx before any response, fall back to a secondary you have evaluated. Never fail over on a 4xx or an ambiguous 408 (the request may already be billed), never switch mid-stream (it splices two answers), and prefer a smaller same-provider model so failover keeps your cost and privacy properties. Prefer the framework primitive (ADK's FallbackModel) over a hand-rolled wrapper: it also restores the request before the next attempt.

Reference implementation

From the AgentOps Open Course (agent modules live under agents/python/src/agent/):

• resilience.py — with_resilience (deadline + bounded retry, never on writes).
• circuit.py — a deterministic, clock-injectable CircuitBreaker.
• model.py — ADK's FallbackModel wrapping a primary and a validated fallback.
• Course chapters 4.5. Guardrails and 5.4. Model Gateway.