cloud-native-distributed-tracing

Distributed Tracing Architecture

Purpose

Make every request traceable across services, queues, and async jobs without unbounded cost. The hard parts are context propagation across non-HTTP boundaries and choosing a sampling design that keeps errors and slow traces while discarding routine ones.

Agent Protocol

Trigger

"distributed tracing", "trace context", "traceparent", "baggage", "tail sampling", "head sampling", "trace kafka", "jaeger", "trace correlation", "span links".

Input Context

• Languages/frameworks, messaging systems (Kafka, SQS, RabbitMQ), service mesh presence.
• Trace volume (spans/s) and backend (Jaeger, Tempo, vendor APM).
• What must be kept (errors, latency outliers, specific tenants).

Output Artifact

Propagation plan, sampling policy, Collector config. No file unless requested.

Response Format

Propagators: {tracecontext,baggage[,b3]}
Sampling: head={ratio} | tail={policies}
Collector topology: {agent -> load-balancing exporter -> tail-sampling gateway -> backend}
Async boundaries: {how context crosses each queue}

Completion Criteria

• Every boundary (HTTP, gRPC, each queue) propagates context.
• Sampling keeps 100% of errors if tail sampling is used.
• Logs carry trace_id/span_id for correlation.

W3C Trace Context

traceparent: 00-<trace-id 32 hex>-<parent-id 16 hex>-<trace-flags 2 hex>

• 00 = version; trace-flags bit 01 = sampled.
• tracestate carries vendor-specific key/values; must be forwarded unchanged when not understood.
• Baggage (baggage header, W3C Baggage) carries application key/values (e.g. tenant.id) to all downstream services. It is not added to spans automatically, is visible to every hop (no secrets/PII), and adds bytes to every request.

Propagation Across Async Boundaries

HTTP and gRPC instrumentation inject/extract automatically. Messaging needs explicit care:

from opentelemetry import trace, propagate
from opentelemetry.trace import SpanKind, Link

tracer = trace.get_tracer("orders")

def publish(producer, order):
    with tracer.start_as_current_span("orders publish", kind=SpanKind.PRODUCER):
        carrier: dict[str, str] = {}
        propagate.inject(carrier)                      # writes traceparent/tracestate/baggage
        headers = [(k, v.encode()) for k, v in carrier.items()]
        producer.produce("orders", value=order.to_json(), headers=headers)

def consume(msg):
    carrier = {k: v.decode() for k, v in (msg.headers() or [])}
    ctx = propagate.extract(carrier)
    # Option A: continue the trace (one consumer per message)
    with tracer.start_as_current_span("orders process", context=ctx, kind=SpanKind.CONSUMER):
        handle(msg)

def consume_batch(msgs):
    # Option B: batch — one span with links to each producer context
    links = [Link(trace.get_current_span(propagate.extract(
        {k: v.decode() for k, v in (m.headers() or [])})).get_span_context()) for m in msgs]
    with tracer.start_as_current_span("orders process batch", links=links, kind=SpanKind.CONSUMER):
        for m in msgs:
            handle(m)

Use span links whenever one span relates to many parents (batches, fan-in) — parent/child only models one.