reliability-design

Reliability design

Read composition guidance for optional expertise, fallback and scope.

Start with what the user depends on and decide what counts as completing it successfully. Define a service-level indicator (SLI) that measures that outcome over a known population, then establish an accepted service-level objective (SLO) and the monitoring needed to assess it. Infrastructure and request metrics help explain failure, but the indicator needs to retain the meaning of the user outcome.

Decide what eligible work enters the denominator before defining success. Include work that starts but never reaches a terminal event, using a deadline or reconciliation observation where necessary. Otherwise the system can appear reliable by counting only work that managed to finish. Read the journey-to-SLO reference below when deriving the measurement or its arithmetic.

Inputs and execution mode

Discover the target, architecture, relevant feature/critical journeys, current telemetry, existing objectives, owners and operational policy. Use code, docs, rendered configuration and live evidence when available. Work with partial sources but label coverage and semantic decisions still needed.

• Interactive: analyze evidence before asking; ask targeted questions about users, acceptable outcomes, latency and budget decisions. Explain alternatives with diagrams and examples.
• Setup/feature implementation: produce a journey contract and implement its instrumentation/resource definitions, tests and documentation in the target stack. Keep changes compatible with existing consumers.
• Recurring review: compare changed flows and incidents with current contracts, identify uncovered outcomes and telemetry drift, update evidence-backed definitions within assigned policy, and prepare bounded patches. Do not automatically relax targets or suppress pages to make compliance improve.

Derivation workflow

1. Identify the users (human or machine), intent/start boundary, observable completion, failure/cancellation/rejection branches, deadline, and important segments.
2. Draw an evidenced architecture and Mermaid sequence. Include browser/client code, cached paths, external providers, retries, async work, and dependencies required for the outcome. Use current-system mapping expertise if discovery is needed, or perform the relevant mapping directly.
3. Classify request-driven, storage and processing stages. Choose a small set of dimensions: availability, latency, quality, freshness, coverage, correctness or durability.
4. Write an SLI specification in outcome terms. Separate technical handling from conversion/business acceptance and infrastructure diagnostics.
5. Define the implementation contract: valid event population, good-event subset, observation point, timer, units, deduplication, retries, terminal/deadline attribution, tags, queries and missing-data behavior.
6. Compare possible observation strategies by user-experience fidelity, coverage, cost, latency and collection reliability. Name blind spots, consent/sampling bias and requests that never reach instrumentation.
7. Choose targets/windows from user needs and measured defensibility. Label initial hypotheses and aspirational targets. Document owner agreement and an error-budget decision policy; targets are product decisions, not arbitrary agent defaults.
8. Define symptom alerts and useful predictive risk alerts. Determine responder/action/runbook, scope, routing, no-data policy, timing and recovery. Prefer multiwindow burn rates where appropriate; validate low-traffic and extreme-target behavior.
9. Implement metric/monitor/SLO/dashboard definitions using the target's tools. Add missing instrumentation with observability-operations or directly. Keep SLO populations consistent with their indicators; label intentionally different operational proxies.
10. Validate with fixtures, historical incidents/quiet periods and live signal samples. Soak new pages silently when local practice permits, with a review date. Confirm notification and runbook reachability before marking operational.