resilience-patterns

Resilience Patterns

Core Principle

Every external call will eventually fail. Resilience is not about preventing failure -- it is about controlling the blast radius and recovering quickly.

---

1. Circuit Breaker

What it does: Stops calling a failing downstream service, giving it time to recover instead of overwhelming it with doomed requests.

States: CLOSED (normal) -> OPEN (failing, reject calls) -> HALF-OPEN (test with limited traffic).

When to apply: Any synchronous call to an external service, database, or third-party API.

TypeScript Example (using cockatiel)

import { CircuitBreakerPolicy, SamplingBreaker, handleAll } from 'cockatiel';

const breaker = new CircuitBreakerPolicy(handleAll, {
  halfOpenAfter: 10_000,       // Try again after 10s
  breaker: new SamplingBreaker({
    threshold: 0.5,            // Open when 50% of requests fail
    duration: 30_000,          // Over a 30s window
    minimumRps: 5,             // Need at least 5 req/s to evaluate
  }),
});

breaker.onBreak(() => console.warn('Circuit OPEN -- downstream is failing'));
breaker.onHalfOpen(() => console.info('Circuit HALF-OPEN -- testing downstream'));
breaker.onReset(() => console.info('Circuit CLOSED -- downstream recovered'));

const result = await breaker.execute(() => callDownstreamService());

Python Example

import circuitbreaker

@circuitbreaker.circuit(failure_threshold=5, recovery_timeout=30)
def call_downstream_service():
    response = requests.get("https://api.partner.com/data", timeout=5)
    response.raise_for_status()
    return response.json()

# Usage with fallback
try:
    data = call_downstream_service()
except circuitbreaker.CircuitBreakerError:
    data = get_cached_or_default_data()

---

2. Bulkhead

What it does: Isolates failures by partitioning resources (thread pools, connection pools, rate limits) so one failing dependency cannot exhaust resources needed by others.

When to apply: A service calls multiple downstream dependencies and you need to prevent one slow dependency from starving others.