observability

Observability and Instrumentation

Code you can't observe is code you can't operate. Instrumentation is written alongside the feature, the same way tests are — if a feature ships without telemetry, the first user-reported bug becomes archaeology instead of a query.

When to use

• Building a feature that will run in production (new service, endpoint, background job, external integration).
• A production incident took too long to diagnose because the system couldn't tell what happened.
• Adding or reviewing alerting rules.
• Reviewing a PR that adds I/O, retries, queues, or cross-service calls.

Not for: Diagnosing a live failure (use debugging) or profiling measured slowness (use performance). Observability is what makes those skills fast next time.

Steps

1. Define "working" before instrumenting

Write down 2–4 questions an on-call engineer will ask about this feature. Every signal you add must answer one of them — if you can't name the questions, you'll log everything and learn nothing.

If the service has no SLO, define one before setting alert thresholds. An SLO turns "is it healthy?" from a feeling into a number — see references/slo-methodology.md for SLI definition, error budgets, and burn-rate alerting.

2. Pick the right signal per question

• Structured log — "what happened in this specific case?" (per-event; grows with traffic)
• Metric — "how often / how fast, in aggregate?" (fixed per series; cheap to query)
• Trace — "where did time go across services?" (per-request; usually sampled)

Rule of thumb: metrics tell you that something is wrong, traces tell you where, logs tell you why.

3. Structured logging

Log events, not prose. Every line is a JSON object with a stable event name and machine-readable fields.

// BAD: string interpolation — unqueryable, inconsistent
logger.info(`Payment ${id} failed for user ${userId} after ${n} retries`);

// GOOD: stable event name + structured fields
logger.warn({ event: 'payment_failed', paymentId: id, provider: 'stripe',
              errorCode: err.code, attempt: n }, 'payment failed');