metrics-logs-traces

Observability: Metrics, Logs, and Traces

The three pillars of observability for understanding system behavior.

The Three Pillars

┌─────────────────────────────────────────────────────────────┐
│                    OBSERVABILITY PILLARS                    │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  METRICS              LOGS                TRACES             │
│  ────────            ─────               ───────             │
│  What's happening    Why it's happening  How requests flow   │
│                                                              │
│  • Request rate      • Error messages    • Request path      │
│  • Error rate        • Stack traces      • Service deps      │
│  • Latency           • Debug info        • Timing breakdown  │
│  • Saturation        • Audit events      • Correlation       │
│                                                              │
│  Aggregated          Individual          Request-scoped      │
│  Time-series         Events              Distributed         │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Metrics: The RED Method

For services, track Rate, Errors, and Duration:

# Example: Prometheus metrics in Python
from prometheus_client import Counter, Histogram, Gauge

# Rate: Request throughput
request_total = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status']
)

# Errors: Error rate
error_total = Counter(
    'http_errors_total',
    'Total HTTP errors',
    ['method', 'endpoint', 'error_type']
)

# Duration: Latency distribution
request_duration = Histogram(
    'http_request_duration_seconds',
    'HTTP request duration',
    ['method', 'endpoint'],
    buckets=[.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]
)

# Usage in handler
@request_duration.labels(method='GET', endpoint='/api/users').time()
def get_users():
    request_total.labels(method='GET', endpoint='/api/users', status='200').inc()
    # ... handler logic