optimizing-metrics-cost

Diagnose metrics cardinality/cost problems, then recommend the safe fix. You can run diagnostic queries yourself via the Prometheus datasource tools; remediation (Adaptive Metrics rules, source fixes) usually belongs to the user.

The One Rule: never drop a label that makes a series unique

Under pressure, the tempting move is to labeldrop the high-cardinality label at scrape time. Do not recommend it. Removing pod, instance, or any label that distinguishes real series silently breaks the data: counter resets from different series interleave → rate()/increase() return garbage (often absurdly high), duplicate samples inflate DPM instead of reducing it, and there is no evidence left of where it broke.

The only safe remediations, in order of preference:

1. Fix the source — stop emitting the bad label (the real fix for unbounded path, user_id).
2. Adaptive Metrics — aggregates correctly (counter-reset-aware, audited, reversible). For Grafana Cloud: portal → Adaptive Metrics → review auto-recommendations sorted by series-reduction impact. Rules apply within ~5 minutes; verify with the queries below.
3. Drop an entire unwanted metric — action: drop on __name__ is safe because it discards whole series, not labels.

Investigation playbook (run these yourself)

# Headline: total active series (Grafana Cloud billed unit)
grafanacloud_instance_active_series
# Self-hosted Prometheus / Mimir:
prometheus_tsdb_head_series

# Growth rate over 7d — more than a few %/day on a stable fleet is a red flag
deriv(prometheus_tsdb_head_series[7d]) * 86400

# Top 10 metrics by series count — usually the whole answer
topk(10, count by (__name__)({__name__=~".+"}))

# Per-metric drill-down: unique values per label on the suspect metric
count by (label_name) (count by (__name__, label_name) (<metric>))

# Top label values inside one label
topk(20, count by (path) (http_requests_total))

# Series-per-instance breakdown — if uneven, one instance is misbehaving
count by (instance) (<metric>)

# Recent change: what did a deploy add? Compare offset 1d/7d, correlate with build_info
count by (__name__)({__name__=~".+"}) - count by (__name__)({__name__=~".+"} offset 1d)