observability

Observability

Iron Law

NO USER-REACHABLE SERVICE PATH SHIPS BLIND.

When to Use

• Logs, metrics, traces, health checks, dashboards, SLOs, alerts, dependency health, incident diagnosis, OpenTelemetry, cardinality, burn-rate alerts.
• Improving production performance; relevant metrics must be in place.

When NOT to Use

• Local-only scripts or libraries with no operational surface.
• Error type design; use error-handling. Release sequencing; use release.

Rules

1. Every new user-reachable path emits request, error, and duration (RED) signals, and traces cover its inbound and outbound boundaries. Critical dependencies expose latency, error, timeout, retry, circuit state, and saturation.
2. When improving production performance, put relevant performance metrics in place before changing behavior and verify their values afterward. Use performance for the baseline and comparable before/after evidence; without that evidence, do not claim an improvement.
3. Logs are structured events with stable names, typed fields, severity, outcome, and trace or correlation ID, using OpenTelemetry semantic conventions where they exist. Levels distinguish expected client failures from operator-actionable errors. Load references/logging.md for any logging change.
4. Metric labels are bounded. User IDs, request IDs, and paths go in logs or traces, because cardinality is a production cost.
5. Sensitive data is redacted at the source with a field allowlist (see security); collector filtering is defense in depth only.
6. Liveness never depends on external systems. Readiness does.
7. Alerts fire on user impact, SLO burn, or actionable dependency failure, and link to a runbook with action and escalation. Tune noisy alerts by signal, owner, or threshold; never silence them.
8. Dashboards answer current health and likely fault location.
9. A prototype may defer observability only if the deferral is recorded and the path is promoted before real users reach it.