logging-and-observability

Observability is the ability to ask a new question about the system's behaviour without shipping new code. Logs, metrics, and traces each answer different questions; a service needs all three, wired so that one request can be followed across them.

Logs

• Structured (JSON, one object per line) to stdout; the platform ships them. Never format strings for humans in production logs; dashboards and queries need fields.
• Standard fields on every line: timestamp (ISO 8601 UTC), level, message, service, version, env, request_id or trace_id, and the domain ids in play (user_id, order_id). A logger bound with context (logger.child({ requestId })) so handlers do not repeat them.
• Levels: error for failures needing attention (paged or triaged), warn for degraded but handled, info for business events and request summaries (one line per request with method, path, status, duration), debug off in production. A log at error that nobody would act on is noise that hides the one that matters.
• One request summary line at the boundary rather than a line per step; steps at debug.
• Never log: secrets, tokens, passwords, full card numbers, full request bodies, personal data beyond the ids needed. Redaction in the logger config (pino redact, structlog processors), not by remembering.
• Sampling for high-volume info and for debug in production when needed; errors are never sampled.
• Retention and cost: logs cost by volume; a noisy dependency at info can be the largest line item. Budget per service and review monthly.

Libraries: pino (Node), structlog or logging with a JSON formatter (Python), slog (Go), tracing (Rust).

Correlation

• A request_id accepted from the inbound header (X-Request-Id, or the W3C traceparent) or generated at the edge, put on the logger context and the response header, and forwarded on every outbound call and every enqueued job. A job carries the id of the request that created it.
• With tracing enabled, the trace id serves as the correlation id; log it on every line.