loom-logging-observability

Logging and Observability

Overview

Understand system behavior through the three pillars — logs, metrics, traces — correlated by shared IDs. This skill covers structured logging, OpenTelemetry tracing, Prometheus metrics, aggregation backends, and alerting, with emphasis on the cost/cardinality traps and sampling decisions that separate a working setup from an expensive broken one.

Three Pillars — what each answers, and its cost model

Pillar	Answers	Cost driver	Use for
Metrics	"Is it broken? how much?" (aggregate)	Label cardinality (# series)	Dashboards, SLOs, alerting — always-on, cheap
Traces	"Where in the request path?" (causal)	Span volume → sampling	Latency breakdown, cross-service dependency
Logs	"What exactly happened?" (event detail)	Volume + indexing strategy	Forensics, audit, the specifics of one request

Reach for metrics first (cheap, aggregate), traces to localize, logs for the detail. Link all three by trace_id/correlation_id so you can pivot: alert fires on a metric → jump to an exemplar trace → read that trace's logs.

Structured Logging

Emit JSON, one object per event — never string-interpolated prose. Structured fields are queryable in any backend; f"user {id} did {action}" is not.

import json, logging, sys
from datetime import datetime, timezone
from contextvars import ContextVar

correlation_id: ContextVar[str] = ContextVar("correlation_id", default="")
trace_id: ContextVar[str] = ContextVar("trace_id", default="")

class JsonFormatter(logging.Formatter):
    def format(self, r: logging.LogRecord) -> str:
        data = {
            "ts": datetime.now(timezone.utc).isoformat(),
            "level": r.levelname, "logger": r.name, "msg": r.getMessage(),
            "correlation_id": correlation_id.get(), "trace_id": trace_id.get(),
        }
        if r.exc_info:
            data["exception"] = self.formatException(r.exc_info)
        if hasattr(r, "fields"):
            data.update(r.fields)          # structured extras
        return json.dumps(data)

h = logging.StreamHandler(sys.stdout); h.setFormatter(JsonFormatter())
logging.getLogger().addHandler(h); logging.getLogger().setLevel(logging.INFO)

logging.getLogger(__name__).info("order processed",
    extra={"fields": {"order_id": order.id, "total": order.total}})