opentelemetry

OpenTelemetry (OTel)

Production standards for instrumenting applications to achieve high-fidelity observability.

1. Distributed Tracing

• Auto-Instrumentation & eBPF: Rely heavily on auto-instrumentation and "OTel Blueprints." Manual spans are reserved specifically for complex business logic.
• Trace Context Propagation: Always propagate the traceparent and tracestate headers across HTTP and RPC boundaries (using W3C Trace Context).
• Span Granularity: Create spans for logical units of work. Avoid creating spans for every single function call (which causes overhead). Focus on:
  • Incoming HTTP requests
  • Database queries
  • GenAI Observability (tracking LLM calls, token exchanges, tool invocations)
  • Background task executions
• Semantic Conventions: Use standardized span attributes (e.g., http.method, http.status_code, db.system). Do not invent custom attribute names when standard ones exist.
• GenAI Semantic Conventions: When tracing LLM interactions, agent orchestration, or Model Context Protocol (MCP) tool calls, you MUST use the official gen_ai.* semantic conventions (e.g., gen_ai.system, gen_ai.request.model, gen_ai.usage.prompt_tokens). This ensures observability vendors (like Datadog) can properly map and analyze AI token usage and latency.

2. Span Implementation Guidelines

• Status and Errors: explicitly set the span status to Error when an exception occurs, and record the exception object on the span.
• Payloads: Avoid logging sensitive PII or massive payloads in span attributes. Log structural identifiers (e.g., user.id, tenant.id).

// Example: Creating a Span in Node.js
tracer.startActiveSpan('database.query', (span) => {
  try {
    span.setAttribute('db.statement', queryText)
    const result = db.execute(queryText)
    span.setStatus({ code: SpanStatusCode.OK })
    return result
  } catch (error) {
    span.recordException(error)
    span.setStatus({ code: SpanStatusCode.ERROR, message: error.message })
    throw error
  } finally {
    span.end()
  }
})