resilience-flow-control

Think Through Resilience & Flow Control

Overview

Production guidance for surviving load and dependency failure. Each reference paper captures the mechanics that separate resilient systems from hopeful ones: retry storms, synchronized expiry, thundering herds, timeouts that compose incorrectly, circuit breakers without fallbacks, and rate limits that fail open exactly when they matter.

Core principle: Every dependency fails, every cache goes stale, and every retry amplifies load. Controls must have explicit semantics (window, scope, fail-open/closed) and the system must stay bounded while capacity is unavailable.

Domain Law

NO RESILIENCE OR FLOW-CONTROL CHANGE WITHOUT:
1. the minimum required primary paper(s) for the control selected from the context table;
2. the failure being defended against named (timeout, overload, staleness,
   provider outage) before choosing a mechanism;
3. "Existing-codebase checks" run when changing existing controls;
4. every applicable MUST mapped to a bounded decision, a test that
   injects the failure, or a documented exception.

When to Use

Use this skill when thinking through, reviewing, changing, or verifying:

• caches: placement, TTL, eviction, invalidation, warming, negative caching, stampede protection;
• distributed cache coordination and hot-key handling;
• rate limiting by IP/user/tenant/endpoint, window algorithms, distributed limiting, fail-open/closed policy;
• quotas: soft/hard limits, reservation, reset windows, overage;
• external integrations: provider contracts, outages, degradation, replacement risk;
• retries: classification, caps, budgets, exponential backoff with jitter, honoring provider pushback;
• timeouts: per-hop and end-to-end deadlines, cancellation propagation;
• circuit breakers: trip thresholds, half-open probing, fallback behavior;
• bulkheads and concurrency limits;
• graceful degradation and load shedding;
• backpressure and admission control for queues and workers.