distributed-tracing

Skill: Distributed Tracing

Expertise: OpenTelemetry SDK, auto-instrumentation, Tempo/Jaeger, trace-log correlation, sampling strategies.

When to load

When adding tracing to a service, debugging slow distributed transactions, or setting up trace → log → metric correlation.

End-to-End Setup Workflow

1. Deploy collector — configure and deploy the OTel Collector as a DaemonSet (see config below)
2. Instrument service — add SDK initialization and auto-instrumentation for your framework (Python/Go examples below)
3. Verify traces — confirm traces appear in Tempo/Jaeger: curl -s http://tempo:3200/api/search?q={}&limit=5
4. Add log correlation — inject trace_id and span_id into log lines for Loki/Grafana linkage
5. Validate linkage — click a trace in Grafana → Explore → verify it links to the corresponding log entries
6. Tune sampling — apply tail-based sampling policies for errors and slow traces (see strategy table)

OpenTelemetry Collector (K8s DaemonSet)

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc: { endpoint: "0.0.0.0:4317" }
      http: { endpoint: "0.0.0.0:4318" }

processors:
  batch:
    timeout: 1s
    send_batch_size: 1000
  memory_limiter:
    check_interval: 1s
    limit_mib: 400

  # Tail-based sampling — sample 100% of error/slow traces
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors-policy
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow-traces
        type: latency
        latency: { threshold_ms: 500 }
      - name: probabilistic-10pct
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls: { insecure: true }

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/tempo]

Python Auto-Instrumentation (FastAPI)

# main.py — add before app creation
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor

provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="otel-collector:4317"))
)
trace.set_tracer_provider(provider)

# Auto-instrument frameworks
FastAPIInstrumentor.instrument_app(app)
HTTPXClientInstrumentor().instrument()  # outgoing HTTP calls
SQLAlchemyInstrumentor().instrument()   # DB queries