opentelemetry

OpenTelemetry

Adopt vendor-neutral telemetry with consistent instrumentation across services.

When to Use This Skill

• Debugging latency across microservices
• Standardizing observability data model and naming
• Sending telemetry to Prometheus, Grafana, Datadog, or OTLP backends
• Building SLO dashboards with trace-to-log correlation
• Instrumenting Python or Node.js applications with tracing and metrics
• Setting up auto-instrumentation for existing services without code changes

Prerequisites

• Application services running in containers or on VMs
• Backend for traces (Jaeger, Tempo, Datadog, or any OTLP receiver)
• Backend for metrics (Prometheus, Mimir, or OTLP receiver)
• Kubernetes cluster (for collector deployment) or VM with systemd
• Network access from services to collector, and collector to backends

Core Workflow

1. Define semantic conventions for services, environments, and versions.
2. Add SDK or auto-instrumentation in each service.
3. Run an OpenTelemetry Collector to receive, transform, and export telemetry.
4. Validate cardinality and sampling to control cost.
5. Create golden signals dashboards and alerting from collected data.

Collector Production Configuration

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

  # Scrape Prometheus endpoints
  prometheus:
    config:
      scrape_configs:
        - job_name: "kubernetes-pods"
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              action: keep
              regex: "true"
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
              action: replace
              target_label: __address__
              regex: (.+)
              replacement: $$1

  # Host metrics for infrastructure monitoring
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu: {}
      memory: {}
      disk: {}
      network: {}
      load: {}

processors:
  batch:
    send_batch_size: 1024
    timeout: 5s

  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128

  attributes:
    actions:
      - key: deployment.environment
        value: production
        action: upsert

  # Drop high-cardinality attributes to control cost
  filter/drop-debug:
    traces:
      span:
        - 'attributes["http.request.header.x-debug"] == "true"'

  # Reduce cardinality on URL paths
  transform/normalize-routes:
    trace_statements:
      - context: span
        statements:
          - replace_pattern(attributes["url.path"], "/users/[0-9]+", "/users/{id}")
          - replace_pattern(attributes["url.path"], "/orders/[0-9]+", "/orders/{id}")

  # Resource detection for cloud environments
  resourcedetection:
    detectors: [env, system, gcp, aws, azure]
    timeout: 5s

exporters:
  # Send traces to Tempo/Jaeger
  otlp/traces:
    endpoint: tempo:4317
    tls:
      insecure: true

  # Send metrics to Prometheus via remote write
  prometheusremotewrite:
    endpoint: http://mimir:9009/api/v1/push
    tls:
      insecure: true

  # Send logs to Loki
  otlp/logs:
    endpoint: loki:4317
    tls:
      insecure: true

  # Debug exporter for development
  debug:
    verbosity: basic

service:
  telemetry:
    logs:
      level: info
    metrics:
      address: 0.0.0.0:8888

  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, transform/normalize-routes, batch, attributes]
      exporters: [otlp/traces]
    metrics:
      receivers: [otlp, prometheus, hostmetrics]
      processors: [memory_limiter, resourcedetection, batch, attributes]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, batch, attributes]
      exporters: [otlp/logs]