structured-logging-and-tracing

Instrument code with structured logs, metrics and traces

Observability is designed where the code knows what happened. Emit events a machine can filter and a human can read, carry one trace context through every hop, measure the few things an alert needs, and never write a secret or a person's data into a log that lives for ninety days. Sources: W3C Trace Context (Recommendation); OpenTelemetry semantic conventions (HTTP spans and metrics, log data model); OWASP Logging Cheat Sheet; Google SRE book ch. 6 (four golden signals).

Steps

1. Use the repo's instrumentation. Find the structured logger, metrics client, tracing SDK (OpenTelemetry if present) and the correlation convention. Call search_lore for standard field names (request_id, ticket, duration_ms). A second logger or tracer is a defect.
2. Log events, not sentences. One record per meaningful event: stable event name plus typed fields, e.g. {"event":"claim.released","ticket":42,"reason":"ttl_expired"}, not "released ticket " + n. Units in field names (duration_ms, size_bytes). The logger's serialiser escapes values, which also stops log injection (CR/LF in user input forging lines); never build log lines by concatenation.
3. Choose levels deliberately. error: a failure someone must act on. warn: degraded but handled. info: state transitions an operator follows. debug: the rest. A swallowed exception is logged with its error class and cause, never at debug only.
4. Propagate trace context across every hop. Accept and emit W3C traceparent (00-<32 hex trace-id>-<16 hex span-id>-<flags>) and tracestate on HTTP; put the same fields in queue message headers and job payloads, and pass them to spawned processes (env or argument). Stamp trace_id and span_id on every log record in that unit of work (OTel log correlation). If the repo also has a request_id, carry both. Do not put secrets or personal data in baggage; it travels to third parties.
5. Add spans at slow or flaky boundaries — inbound requests, outbound HTTP, DB queries, queue publish/consume, subprocesses — using the SDK's auto-instrumentation where it exists. Follow semantic conventions: HTTP span name {method} {route} (GET /tickets/{id}, never the raw URL with ids), attributes http.request.method, http.route, http.response.status_code, server.address, error.type. Status: server spans mark 5xx as Error and leave 4xx unset; client spans mark both 4xx and 5xx as Error. Record the exception on the span; never attach payloads.
6. Measure what an alert would use. The golden signals for the operation: a duration histogram (OTel: http.server.request.duration, seconds), a counter of outcomes by result, and saturation gauges (queue depth, pool in-use). Metric labels are low-cardinality (route template, status class, kind); user ids, ticket numbers, URLs and error messages go on spans or logs, never labels. Alert design: the observability-designer skill.
7. Log the security-relevant events (OWASP): authentication success and failure, authorisation denials, input-validation failures at trust boundaries, admin and permission changes, with who, what, when, from where and outcome.
8. Redact by construction. Never log passwords, tokens, session ids, API keys, Authorization/Cookie headers, full request or response bodies, or personal data beyond need. Allow-list emitted fields or use the repo's redaction helper; hash an identifier if correlation is needed (the data-privacy skill).