golang-observability-opentelemetry

Go Observability with OpenTelemetry

Overview

Modern Go applications require comprehensive observability through the three pillars: traces, metrics, and logs. OpenTelemetry provides vendor-neutral instrumentation for distributed tracing, Prometheus offers powerful metrics collection, and Go's slog package (1.21+) delivers structured logging with minimal overhead.

Key Features:

• 🔍 OpenTelemetry: Distributed tracing with context propagation
• 📊 Prometheus: Metrics collection with /metrics endpoint
• 📝 Structured Logging: slog with JSON formatting and correlation IDs
• 🎯 Auto-Instrumentation: HTTP/gRPC middleware patterns
• 💚 Health Checks: Kubernetes-ready readiness/liveness probes
• 🔄 Graceful Shutdown: Clean exporter shutdown and signal handling

When to Use This Skill

Activate this skill when:

• Instrumenting microservices for production observability
• Setting up distributed tracing across service boundaries
• Creating operational dashboards with Prometheus/Grafana
• Debugging production performance issues or bottlenecks
• Implementing SLOs and monitoring SLIs
• Adding observability to existing Go applications
• Correlating logs, traces, and metrics for debugging

Core Observability Principles

The Three Pillars

1. Traces: Understand request flow across distributed systems
2. Metrics: Measure system behavior and performance over time
3. Logs: Record discrete events for debugging and audit

Correlation Strategy

All three pillars must share common identifiers:

• Trace ID: Links all operations in a request
• Span ID: Identifies specific operation within trace
• Request ID: Correlates logs with traces and metrics

OpenTelemetry Integration

Installation

go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/sdk
go get go.opentelemetry.io/otel/exporters/jaeger
go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

Basic Setup

package main

import (
    "context"
    "log"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/jaeger"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
)

func initTracer(serviceName string) (*sdktrace.TracerProvider, error) {
    // Create Jaeger exporter
    exporter, err := jaeger.New(jaeger.WithCollectorEndpoint(
        jaeger.WithEndpoint("http://localhost:14268/api/traces"),
    ))
    if err != nil {
        return nil, err
    }

    // Create resource with service name
    res, err := resource.Merge(
        resource.Default(),
        resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName(serviceName),
            semconv.ServiceVersion("1.0.0"),
        ),
    )
    if err != nil {
        return nil, err
    }

    // Create tracer provider
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(res),
        sdktrace.WithSampler(sdktrace.AlwaysSample()), // Use probability sampler in production
    )

    otel.SetTracerProvider(tp)
    return tp, nil
}

func main() {
    tp, err := initTracer("order-service")
    if err != nil {
        log.Fatal(err)
    }
    defer func() {
        if err := tp.Shutdown(context.Background()); err != nil {
            log.Printf("Error shutting down tracer: %v", err)
        }
    }()

    // Application code...
}