resilience-testing

Test application resilience

Verify the promised user outcome when a relevant dependency or component fails, then verify recovery. Use existing test tools and the smallest boundary that answers the question. Keep tests beside the affected application; no chaos service, MCP or Kubernetes operator is required. Use available journey-testing for demand/load claims and a matching technical specialist for the selected tool. Combine load with a fault only when that interaction is the intended experiment.

Define the hypothesis and boundary

Read the journey and identify a dependency whose failure could prevent or corrupt its outcome. Decide which state of failure matters: unavailable before work starts, interrupted while work is in progress, or unreachable after work may already have completed. Those conditions require different observations and can expose different recovery behavior.

Read the dependency/worker contracts, accepted deadlines and recovery expectations. Explain what the application should tell the caller, what work may remain unfinished, and how that work will be accounted for after recovery. Use reliability, observability and recovery guidance when available.

• State baseline behavior, one failure condition, expected degraded outcome, unaffected paths and the observation that would disprove the hypothesis.
• Derive limits from accepted requirements; label provisional values.
• Do not inject faults solely because a tool supports them or prescribe every scenario for a small feature.

Examples illustrate different failure boundaries:

Failure	Outcome to establish
Slow provider	A bounded error while unrelated reads work.
Worker restart	Accepted work resumes without repeating a business effect.
Loss of database access	Mutations fail without corrupting data and normal operation returns afterward.

HTTP success, pod readiness and a green alert alone do not prove these outcomes.

Choose the least intrusive method

Method	Useful claim	Limit
Deterministic fixtures/fake clients	Timeouts, malformed responses, transient/permanent errors, retry limits, authorization and failure presentation	Application handling of simulated failures, not actual network/provider behavior.
Isolated application/CI stack	Owned worker termination, controlled dependency unavailability, delayed connections and bounded pool exhaustion with real selected persistence	That stack and fault mechanism, not regional or fleet-wide availability.
Named non-production deployment	Actual routing, process lifecycle, collection and recovery interactions	That configuration and authorized scope; shared users and dependencies still matter.