monitoring-ops

Monitoring Operations

Comprehensive observability patterns covering the three pillars (metrics, logging, tracing), alerting strategies, dashboard design, and infrastructure monitoring for production systems.

---

Three Pillars Quick Reference

Use this table to decide which observability signal fits your need:

Pillar	Best For	Tools	Data Type
Metrics	Aggregated numeric measurements, trends, alerting on thresholds	Prometheus, Datadog, CloudWatch, StatsD	Time-series (numeric)
Logs	Discrete events, error details, audit trails, debugging context	Loki, ELK, CloudWatch Logs, Fluentd	Unstructured/structured text
Traces	Request flow across services, latency breakdown, dependency mapping	Jaeger, Tempo, Zipkin, Datadog APM	Span trees (structured)

When to use which:

• "How many requests per second?" → Metrics (counter + rate)
• "Why did this specific request fail?" → Logs (error message + stack trace)
• "Where is the latency in this request?" → Traces (span waterfall)
• "Is the system healthy right now?" → Metrics (gauges + alerts)
• "What happened at 3:42 AM?" → Logs (timestamped event search)
• "Which downstream service caused the timeout?" → Traces (span analysis)

Correlation is key: Connect all three by embedding trace_id in log entries, recording exemplars in metrics, and linking trace spans to log queries.

---

Metrics Type Decision Tree

Use this tree to select the correct metric type:

What are you measuring?
│
├─ A count of events that only goes up?
│  └─ COUNTER
│     Examples: http_requests_total, errors_total, bytes_sent_total
│     Use rate() or increase() to get per-second or per-interval values
│     Never use a counter's raw value — it resets on restart
│
├─ A current value that goes up AND down?
│  └─ GAUGE
│     Examples: temperature_celsius, active_connections, queue_depth
│     Use for snapshots of current state
│     Can use avg_over_time(), max_over_time() for trends
│
├─ A distribution of values (latency, size)?
│  │
│  ├─ Need aggregatable quantiles across instances?
│  │  └─ HISTOGRAM
│  │     Examples: http_request_duration_seconds, response_size_bytes
│  │     Define buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
│  │     Use histogram_quantile() for percentiles (p50, p95, p99)
│  │     Aggregatable across instances (histograms can be summed)
│  │
│  └─ Need pre-calculated quantiles on a single instance?
│     └─ SUMMARY
│        Examples: go_gc_duration_seconds
│        Pre-calculates quantiles client-side
│        NOT aggregatable across instances
│        Prefer histogram unless you have a specific reason
│
└─ None of the above?
   └─ INFO metric (labels only, value=1)
      Examples: build_info{version="1.2.3", commit="abc123"}
      Use for metadata exposed as metrics