observability-setup

Observability Setup

Overview

Observability is the ability to ask arbitrary questions about a system's behavior from the outside, using its telemetry. This skill instruments a service across the three pillars — structured logs, metrics, and distributed traces — then turns raw signals into SLIs, SLOs, error budgets, and actionable alerts. The goal is not "more dashboards" but the ability to answer "is the user in pain, and where?" in under five minutes.

Keywords: observability, OpenTelemetry, OTel, structured logging, JSON logs, Prometheus, metrics, RED method, USE method, distributed tracing, spans, trace context, correlation ID, SLI, SLO, error budget, burn rate, alerting, Alertmanager, Grafana, on-call, golden signals, cardinality, exemplars.

Standardize on OpenTelemetry (OTel) for instrumentation — it is vendor-neutral and emits to Prometheus, Tempo, Loki, Datadog, Honeycomb, etc. Keep instrumentation code free of vendor SDKs; push vendor choice to the OTel Collector.

Workflow

1. Inventory the service. Identify language/runtime, request entry points (HTTP/gRPC/queue), critical user journeys, downstream dependencies, and the existing telemetry backend (or pick one). Read references/instrumentation-patterns.md for per-pillar guidance.
2. Add structured logging first. Emit one JSON event per request with a stable schema (timestamp, level, message, trace_id, span_id, service, plus event-specific fields). Never log secrets/PII. Logs become useful only once they are queryable and correlated to traces.
3. Instrument metrics using RED + USE. For request-driven services apply RED (Rate, Errors, Duration); for resources apply USE (Utilization, Saturation, Errors). Use the four golden signals (latency, traffic, errors, saturation) as the checklist. Watch label cardinality.
4. Add distributed tracing. Auto-instrument the framework, propagate W3C traceparent across service boundaries, and add manual spans around meaningful business operations. Attach trace_id to every log line and metric exemplar so you can pivot signal → signal.
5. Wire the OTel Collector. Run a collector (agent + gateway) to receive OTLP, batch, sample, and export. This decouples app code from backends and centralizes sampling/redaction. See references/instrumentation-patterns.md.
6. Define SLIs and SLOs. Pick 1–3 SLIs per user journey (availability, latency, correctness). Set realistic SLO targets and a measurement window. Use references/slo-cookbook.md and the template in templates/slo-spec.md.
7. Compute error budgets and burn-rate alerts. Alert on budget burn rate, not on raw thresholds. Use multi-window multi-burn-rate alerts to balance fast detection against false pages. Generate rules with scripts/gen_slo_alerts.py.
8. Validate and reduce noise. Run scripts/check_slo_burn.py against live metrics to sanity-check. Ensure every alert is actionable, links to a runbook, and pages a human only when users are affected. Everything else is a ticket or a dashboard.