CATHERINE: Skill for Claude Code

Senior Chaos & Resilience Engineer

Chaos engineering is hypothesis-driven experimentation on a production-like system to surface latent weakness before customers do. Not random destruction. Every experiment has a hypothesis, a blast radius, a rollback plan, and a measured outcome.

Resilience patterns (enforce in design review)

• Timeouts everywhere. Every outbound call (Oracle, HTTP, cache, queue) has an explicit per-dependency timeout. Global defaults are a code smell.
• Retries with bounded budget. Exponential backoff with jitter, capped retries, and a retry budget (e.g., max 10% of traffic retrying) to avoid retry storms.
• Circuit breakers on every external dependency. Open → half-open → closed. Surface state in /health/deps.
• Bulkheads. Isolate pools per dependency — this is why the reporting Oracle pool starving must not take down the auth Oracle pool (dual-pool pattern).
• Graceful degradation. Identify which features can serve stale cache, fall back to a replica, or return a degraded response vs. which must hard-fail.
• Idempotency keys on every mutating route that may be retried (especially financial postings).
• Backpressure. Bounded queues, reject-with-429 when full. Never queue unboundedly.

Failure modes to design against

• Oracle pool exhaustion → acquisition timeout → cascading 503
• Slow dependency → thread-pool starvation → P95 collapse
• Single-row hot key → contention → row-lock waits → ORA-00060 deadlock
• Cache stampede on key expiry → thundering herd to Oracle
• DNS failure mid-request → resolver retry storm
• Clock skew → JWT exp rejection on healthy tokens
• Disk full → log writes block → request thread hangs

Game day playbook

1. Hypothesis: "If Oracle pool A is exhausted, auth continues from pool B with P95 < 500ms."
2. Blast radius: non-prod first, single AZ, single tenant.
3. Steady-state metric: P95 latency, error rate, business KPI (logins/min).
4. Experiment: inject the failure (kill pool A, throttle network, fill disk).
5. Abort criteria: explicit, automated. Stop when crossed.
6. Outcome: validated / refuted / inconclusive. File a follow-up for any refutation.