observability-patterns

Observability Patterns

Structured Logging

Python (structlog)

import structlog

logger = structlog.get_logger()

logger.info("user_created", user_id=user.id, email=user.email, source="api")
logger.error("payment_failed", order_id=order.id, error=str(e), amount=amount)

Node.js (Pino)

import pino from "pino";

const logger = pino({ level: "info", transport: { target: "pino-pretty" } });

logger.info({ userId: user.id, action: "login" }, "User logged in");
logger.error({ err, orderId: order.id }, "Payment processing failed");

Log Levels

Level	Use For
error	Failures requiring attention
warn	Unexpected but handled situations
info	Business events, state transitions
debug	Development diagnostics

Rules

• Always use structured key-value pairs, not string interpolation
• Include correlation IDs for request tracing
• Never log sensitive data (passwords, tokens, PII)
• Log at boundaries: API entry/exit, external calls, state changes

OpenTelemetry

Python Setup

from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317"))
)

tracer = trace.get_tracer(__name__)
meter = metrics.get_meter(__name__)

request_counter = meter.create_counter("http_requests_total", description="Total HTTP requests")
request_duration = meter.create_histogram("http_request_duration_seconds")

@tracer.start_as_current_span("process_order")
def process_order(order_id: str):
    request_counter.add(1, {"method": "POST", "endpoint": "/orders"})
    with tracer.start_as_current_span("validate_order"):
        validate(order_id)
    with tracer.start_as_current_span("charge_payment"):
        charge(order_id)