correlate-observability-signals

Correlate Observability Signals

Connect metrics, logs, and traces for unified debugging across the three pillars of observability.

When to Use

• Investigating complex incidents that span multiple systems
• Reducing MTTR (mean time to resolution)
• Building unified observability dashboards
• Implementing distributed tracing
• Moving from siloed tools to unified observability

Inputs

• Required: Prometheus (metrics)
• Required: Log aggregation system (Loki, Elasticsearch, CloudWatch)
• Required: Distributed tracing backend (Tempo, Jaeger, Zipkin)
• Optional: Grafana for unified visualization
• Optional: OpenTelemetry instrumentation

Procedure

See Extended Examples for complete configuration files and templates.

Step 1: Implement Trace Context Propagation

Add trace IDs to all logs and metrics using OpenTelemetry:

// Go example: Propagate trace context to logs
package main

import (
    "context"
    "log"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/trace"
)

func handleRequest(ctx context.Context, userID string) {
    // Extract trace context
    span := trace.SpanFromContext(ctx)
    traceID := span.SpanContext().TraceID().String()

    // Include trace ID in structured logs
    log.Printf("trace_id=%s user_id=%s action=process_request", traceID, userID)

    // Business logic here
    processData(ctx, userID)
}

func processData(ctx context.Context, userID string) {
    tracer := otel.Tracer("my-service")
    ctx, span := tracer.Start(ctx, "processData")
    defer span.End()

    traceID := span.SpanContext().TraceID().String()
    log.Printf("trace_id=%s user_id=%s action=process_data", traceID, userID)

    // More work
}

Python example:

# Python: Flask with OpenTelemetry
from flask import Flask, request
from opentelemetry import trace
from opentelemetry.instrumentation.flask import FlaskInstrumentor
import logging

app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)

logging.basicConfig(
    format='%(asctime)s trace_id=%(otelTraceID)s span_id=%(otelSpanID)s %(message)s',
    level=logging.INFO
)

@app.route('/api/users/<user_id>')
def get_user(user_id):
    span = trace.get_current_span()
    trace_id = format(span.get_span_context().trace_id, '032x')

    logging.info(f"Fetching user {user_id}", extra={
        'otelTraceID': trace_id,
        'otelSpanID': format(span.get_span_context().span_id, '016x')
    })

    # Business logic
    return {"user_id": user_id}