distributed-tracing-rails

Distributed Tracing in Rails

When a request crosses 4 services, logs in 4 places, traces are the only way to follow it. OpenTelemetry is the vendor-neutral standard — instrument once, export anywhere. This skill sets up production-grade tracing for Rails with minimal noise.

The opinion

Use OpenTelemetry (OTel) — the CNCF standard, vendor-neutral, replaces vendor-specific APM agents. Use auto-instrumentation for Rails, Net::HTTP, Faraday, Sidekiq, Active Record. Propagate context across HTTP (W3C traceparent header), Sidekiq, and Kafka. Sample at the head (parent-based) by default; tune down for high-traffic services. Export to OTLP and let a collector fan out to your backend (Tempo, Honeycomb, Datadog).

Setup

# Gemfile
gem "opentelemetry-sdk"
gem "opentelemetry-exporter-otlp"
gem "opentelemetry-instrumentation-all"

# config/initializers/opentelemetry.rb
require "opentelemetry/sdk"
require "opentelemetry/exporter/otlp"
require "opentelemetry/instrumentation/all"

OpenTelemetry::SDK.configure do |c|
  c.service_name = ENV.fetch("OTEL_SERVICE_NAME", "rails-app")
  c.resource = OpenTelemetry::SDK::Resources::Resource.create(
    "service.version" => ENV.fetch("APP_VERSION", "unknown")
  )
  c.use_all  # turn on every instrumentation we have a gem for
end

Set via env:

• OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
• OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,service.namespace=ecommerce
• OTEL_TRACES_SAMPLER=parentbased_traceidratio
• OTEL_TRACES_SAMPLER_ARG=0.1 # 10% sampling

What use_all instruments

• Rails (controllers, views, mailers)
• Active Record (SQL queries)
• Net::HTTP, Faraday, HTTPClient (outbound HTTP)
• Sidekiq, Active Job (queue + perform)
• Rack
• Redis
• Tilt (template render)
• AwsSdk, MysqlClient, PG, Postgres

Each becomes a span. Auto-context-propagation works for HTTP and Active Job out of the box.

Pattern 1: Custom spans

class PaymentProcessor
  TRACER = OpenTelemetry.tracer_provider.tracer("payment_processor", "1.0")

  def charge(order)
    TRACER.in_span("charge_order", attributes: { "order.id" => order.id, "order.cents" => order.total_cents }) do |span|
      result = Stripe::PaymentIntent.create(...)
      span.set_attribute("stripe.payment_intent_id", result.id)
      result
    rescue Stripe::CardError => e
      span.record_exception(e)
      span.status = OpenTelemetry::Trace::Status.error("declined")
      raise
    end
  end
end