Observability with Prometheus & Grafana

Observability with Prometheus & Grafana

Overview

Master production observability with Prometheus and Grafana - the industry-standard monitoring stack for cloud-native applications. Learn metrics collection, PromQL query language, dashboard design, alerting, and AI-powered anomaly detection (Grafana AI Observability 2024).

When to Use This Skill

• Monitoring production applications and infrastructure
• Implementing SLOs (Service Level Objectives) and SLIs
• Creating custom metrics for business KPIs
• Setting up alerting for proactive incident response
• Debugging performance issues with metrics analysis
• Tracking API latency, error rates, and throughput
• Monitoring AI/ML model performance in production

Core Principles

1. The Four Golden Signals (Google SRE)

# Always monitor these four metrics for every service:

# 1. Latency - How long requests take
http_request_duration_seconds_bucket{le="0.1", job="api"} 8500
http_request_duration_seconds_bucket{le="0.5", job="api"} 9800
http_request_duration_seconds_sum{job="api"} 2450
http_request_duration_seconds_count{job="api"} 10000

# 2. Traffic - How many requests
http_requests_total{method="GET", status="200"} 50000

# 3. Errors - How many requests fail
http_requests_total{method="POST", status="500"} 150

# 4. Saturation - How "full" is the service
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.2

2. Metric Types

from prometheus_client import Counter, Gauge, Histogram, Summary, Info

# Counter - Monotonically increasing (requests, errors)
request_count = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status']
)
request_count.labels(method='GET', endpoint='/api/users', status='200').inc()

# Gauge - Can go up or down (memory usage, queue size)
active_connections = Gauge(
    'active_database_connections',
    'Number of active database connections'
)
active_connections.set(25)
active_connections.inc()  # Increment
active_connections.dec()  # Decrement

# Histogram - Track distributions (latency, request sizes)
request_duration = Histogram(
    'http_request_duration_seconds',
    'HTTP request duration',
    buckets=[0.01, 0.05, 0.1, 0.5, 1.0, 5.0]  # Define buckets
)
with request_duration.time():
    process_request()

# Summary - Similar to histogram, calculates quantiles
response_size = Summary(
    'http_response_size_bytes',
    'HTTP response size in bytes'
)
response_size.observe(1024)

# Info - Static metadata
app_info = Info('app_version', 'Application version info')
app_info.info({'version': '1.2.3', 'environment': 'production'})