observability

Observability

Design observability from operational questions and user-visible outcomes. Do not add logs, metrics, or spans merely to satisfy a three-pillars checklist. Each signal must support a diagnostic, reliability, security, or business decision at an acceptable cost.

Observability Contract

• Start with critical journeys, SLIs, SLOs, failure modes, and the questions an operator must answer during an incident.
• Use metrics for bounded aggregate behavior, traces for request and dependency paths, and logs for discrete events requiring detailed context.
• Emit structured logs. For container streams, prefer one JSON object per physical line on stdout or stderr; let the platform collect and rotate logs.
• Propagate W3C trace context across process boundaries and correlate logs with valid trace and span identifiers when context exists.
• Keep metric labels and span names bounded. Never use user IDs, request IDs, email addresses, raw URLs, or other unbounded values as metric labels.
• Treat telemetry as sensitive data. Minimize, classify, redact, retain, and authorize it deliberately.
• Keep security audit events separate from ordinary application logs when integrity, access, or retention requirements differ.
• Bound telemetry queues, retries, memory, disk, network, and cost. Define which data may be sampled or dropped under pressure.
• Alert on actionable user impact or imminent budget exhaustion. Every page needs an owner, urgency, evidence, and runbook.
• Instrument the telemetry pipeline itself so loss, refusal, delay, and cost are visible.

Workflow

1. Define Questions and Reliability Targets

State:

• critical user journeys and owners;
• availability, latency, correctness, freshness, and durability SLIs;
• SLO windows and error budgets;
• known failure and overload modes;
• forensic or regulatory requirements;
• incident questions the current system cannot answer.

Do not begin with a vendor, agent, collector, dashboard, or list of fields.