resilience

Resilience

Overview

Make I/O failures boring: set explicit timeouts, retry safely, keep operations idempotent, and prevent cascades with circuit breakers and bulkheads.

Enterprise systems fail partially (timeouts, 5xx, queue lag). Resilience patterns make those failures bounded and observable.

Inputs / Outputs

Inputs: I/O boundary code (HTTP/gRPC/DB/cache/queue clients); dependency map (what calls what). Outputs: Hardened code with resilience patterns applied (timeouts, retries, idempotency, breakers, bulkheads); failure model documentation. Consumed by testing and finish.

Workflow

1. Identify the I/O boundary: HTTP, gRPC, DB, cache, queue/stream, third-party API.

2. Define the failure model:

   • which failures are expected/transient vs permanent
   • what “success” means under degradation (fallback? partial data? fail fast?)

3. Apply patterns in this order:

   1. Timeouts + cancellation
   2. Idempotency (especially if retries exist)

   GATE: If adding retries, idempotency (pattern 2) MUST be addressed first. Never retry a non-idempotent operation without an idempotency key or server-side dedupe.

   1. Retries with backoff + jitter (bounded)
   2. Circuit breaker (when a dependency is unhealthy)
   3. Bulkheads / concurrency limits (to protect your own resources)

4. Add observability (retry counts, breaker state, queue lag, error codes). Each metric should have a named decision it supports and an owner — see observability.

5. Add consumer-visible tests for semantics; add a local smoke test for failure modes.

Minimum viable execution

When context or time is constrained, these are the load-bearing steps:

1. Identify I/O boundaries (step 1) — what calls what.
2. Apply timeout + cancellation (step 3.1) — mandatory for every outbound call.
3. Ensure idempotency before retries (step 3.2-3.3) — never retry without it.
4. Verify (step 5) — test that timeout errors are stable and retries are bounded.