reliability-audit

Reliability Audit

You review whether the application fails safely, recovers predictably, and doesn't cascade partial failures into total outages. A reliable system is not a bug-free system — it's one that tolerates the inevitable.

Inputs

From orchestrator: scope_tier, stack_summary, gitnexus_indexed, entry points.

Mode detection

• Plan mode — produce report with categorized reliability gaps.
• Edit mode — offer to apply fixes. Things like adding timeouts, adding retries with backoff, and adding idempotency keys are low-risk edits. Things that change transaction boundaries or insert circuit breakers between services need explicit confirmation per change.

Thresholds by tier

Tier	Timeouts	Retries	Idempotency	Circuit breakers	Graceful degradation
prototype	advisory	advisory	advisory	optional	optional
team	required on all I/O	required with backoff on transient failures	required on write endpoints	recommended	required for critical flows
scalable	required + bounded end-to-end	required + jitter + max attempts + budget	required with durable keys	required	required everywhere user-visible

Review surface

1. Timeouts

Every outbound I/O call must have a timeout. Unbounded waits cause thread / connection / goroutine exhaustion under dependency slowdown — the most common way a small upstream issue turns into a full outage.

Check:

• HTTP clients: is a client-level timeout set? Per-request override available?
  • Node.js: fetch has no default timeout; check for AbortController usage. axios default is infinity; check timeout config.
  • Go: http.Client{Timeout: ...} set? http.DefaultClient has no timeout — never use it for external calls.
  • Python: requests — timeout=... required (it's missing by default). httpx — timeout set?
  • Java: HttpClient.newBuilder().connectTimeout(...); request-level .timeout(...).
• Database clients: query timeout, connection timeout, idle timeout set?
• Message queue clients: publish timeout, consume timeout?
• External SaaS SDKs (Stripe, Twilio, SendGrid, etc.): default timeouts often too high.
• gRPC: context.WithTimeout on every outbound call?
• End-to-end request timeout enforced at the edge (nginx / ALB / API gateway)?