tracing-expert

TRACING-EXPERT -- Distributed Tracing & Observability Specialist

Domain: OpenTelemetry / Distributed Tracing / Span Context / Sampling / Jaeger / Tempo / Correlation

Core Concepts

Trace:  End-to-end journey of a request across services
Span:   Single unit of work within a trace (has start time, duration, status)
Context: trace_id + span_id + trace_flags, propagated across boundaries
Parent: The span that initiated this span
Root:   The first span in a trace (no parent)

Trace
 |-- Span A (API Gateway, 250ms)
      |-- Span B (Auth Service, 15ms)
      |-- Span C (Order Service, 200ms)
           |-- Span D (Database Query, 50ms)
           |-- Span E (Payment API call, 120ms)

OpenTelemetry Setup (Node.js)

// tracing.ts -- Load BEFORE any other imports
import { NodeSDK } from '@opentelemetry/sdk-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'
import { Resource } from '@opentelemetry/resources'
import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from '@opentelemetry/semantic-conventions'

const sdk = new NodeSDK({
  resource: new Resource({
    [ATTR_SERVICE_NAME]: 'order-service',
    [ATTR_SERVICE_VERSION]: '1.2.0',
    'deployment.environment': process.env.NODE_ENV,
  }),
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4318/v1/traces',
  }),
  instrumentations: [getNodeAutoInstrumentations({
    '@opentelemetry/instrumentation-http': {
      ignoreIncomingPaths: ['/health', '/ready', '/metrics'],
    },
    '@opentelemetry/instrumentation-express': { enabled: true },
    '@opentelemetry/instrumentation-pg': { enabled: true },
    '@opentelemetry/instrumentation-redis': { enabled: true },
  })],
})

sdk.start()
process.on('SIGTERM', () => sdk.shutdown())

Manual Instrumentation

import { trace, SpanKind, SpanStatusCode } from '@opentelemetry/api'

const tracer = trace.getTracer('order-service', '1.0.0')

async function processOrder(order) {
  return tracer.startActiveSpan('processOrder', {
    kind: SpanKind.INTERNAL,
    attributes: {
      'order.id': order.id,
      'order.item_count': order.items.length,
      // NEVER put PII (email, name, address) in span attributes
    },
  }, async (span) => {
    try {
      const validated = await tracer.startActiveSpan('validateOrder', async (childSpan) => {
        const result = await validateOrder(order)
        childSpan.setAttribute('validation.passed', result.valid)
        childSpan.end()
        return result
      })

      if (!validated.valid) {
        span.setStatus({ code: SpanStatusCode.ERROR, message: 'Validation failed' })
        span.recordException(new Error(validated.reason))
        return { error: validated.reason }
      }

      const payment = await processPayment(order)
      span.addEvent('payment_processed', { 'payment.id': payment.id })

      span.setStatus({ code: SpanStatusCode.OK })
      return { success: true, payment }
    } catch (error) {
      span.setStatus({ code: SpanStatusCode.ERROR, message: error.message })
      span.recordException(error)
      throw error
    } finally {
      span.end()
    }
  })
}