observability

Observability

Contract

Inputs

Required: the target source files or endpoints to instrument, and their runtime (language and framework). Optional: existing logging, metrics, or tracing libraries; a pinned metrics or tracing backend; SLOs or historical latency and error data for threshold justification; the alert delivery channel and runbook location. When no backend is pinned, write against the vendor-neutral OpenTelemetry APIs so the exporter can be configured later.

Procedure

1. Read the named target code and confirm its runtime and write surface. If the target cannot be identified, stop without writing. Done when: the target code is read and its runtime and write surface are confirmed.
2. Write down 2-4 on-call questions for the feature (for example: what fraction of attempts succeed on the first try; why does a permanent failure happen; is the provider slower than usual). Every signal added below must answer one of these questions and map to a decision an operator will act on; reject any signal that does not. If no question can be named, stop and report; do not instrument. Done when: 2-4 on-call questions are written, each mapping to at least one signal below, and every signal maps to an operator action.
3. Map each question to one signal: how often or how fast in aggregate, a metric; where time goes across services, a trace; what happened in one specific case, a log. Instrument RED (rate, errors, duration) on every request-driven endpoint and external dependency; instrument USE (utilization, saturation, errors) on queues, pools, and hosts. Done when: each question is mapped to a signal type with RED/USE coverage applied.
4. Add structured logging: every line is a JSON object with a stable event name and machine-readable fields (IDs, provider, error code, attempt count). Never interpolate values into prose strings. Use levels consistently: error for broken invariants needing investigation, warn for degraded but handled, info for significant business events, debug off in production by default. Done when: structured JSON logging is added with consistent levels and no prose interpolation.
5. Generate or accept a request ID at the system boundary (for example the x-request-id header, else a UUID), attach it to every log line, span, and outbound call, and echo it on the response. Without it a single request cannot be reconstructed from interleaved logs. Done when: a request ID is generated at the boundary and propagated to every log line, span, and outbound call.
6. Never log secrets, tokens, passwords, or unredacted PII. Allowlist logged fields; never log whole request bodies. Done when: secret and PII logging is prevented with an allowlist.
7. Add metrics: a latency histogram per endpoint and dependency (for example http_request_duration_seconds, buckets spanning roughly 0.05s to 5s, labels method, route template, and status_class holding 2xx/5xx classes, never the raw status code). Read p50/p95/p99, never averages. Labels come only from small fixed sets (route template, status class, provider name); never user IDs, emails, request IDs, full URLs, or error message text. Done when: latency histograms with bounded labels are added per endpoint and dependency.
8. Add tracing: enable OpenTelemetry auto-instrumentation for HTTP, gRPC, and database clients, initialized before application code, with the service name set. Add manual spans only around meaningful internal units of work, carrying the attributes on-call will filter by. Keep span count to the minimum that reconstructs the critical request path. Propagate context across every async boundary (HTTP headers, queue message metadata) or the trace dies at the gap. Sample head-based at a low rate; keep all errors via tail sampling when the backend supports it. Done when: tracing is enabled with auto-instrumentation, minimum manual spans for the critical path, and context propagation across async boundaries.
9. Define symptom-based alerts on what users feel: sustained error rate over a small percentage, p99 latency over seconds, queue age over minutes. Never alert on causes like CPU, pod restarts, or disk usage. Each alert must be actionable (if the response is to ignore it, delete it), link a runbook stating its meaning, first query, and escalation path, carry a threshold and duration justified by the SLO or historical data, and use exactly two severities: page (user-facing, act now) and ticket (degradation, act this week). Done when: symptom-based alerts are defined with runbook links, justified thresholds, and exactly two severities.
10. Build the dashboard around the failure modes revealed by the chosen signals and traces. Every panel must map to a failure mode or an on-call question; omit charts no one will read. Done when: every dashboard panel maps to a failure mode and no unread panel remains.
11. Keep only the surface the system will maintain: remove any instrumentation, label, span, or panel that duplicates another or that no owner will keep current. Done when: no duplicate or unowned surface remains.
12. Verify the surface compiles, loads, and emits locally. Run the system or a representative test and confirm: structured logs appear as JSON with the correlation ID intact (no [object Object]); metric series appear with expected labels and sane values; one request traces end-to-end with no broken spans. If the system cannot be run locally, report that local emission is unverified. Done when: the surface compiles, loads, and emits locally, and every local check passes.