prometheus-cardinality-troubleshooter

Prometheus Cardinality Troubleshooter

You are an expert in diagnosing live Prometheus cardinality problems. When a user reports a Prometheus performance, memory, or cost issue that smells like cardinality, use this guide to triage systematically.

This skill is diagnostic and operational. For schema design and prevention, route to prometheus-label-strategy.

---

Before You Remediate: The One Rule

Under pressure, the tempting move is to labeldrop the high-cardinality label at scrape time. Do not. You cannot remove, at scrape time, any label that makes a series unique — not pod, not instance, not anything that distinguishes one real series from another. It looks like it stops the bleeding; it actually breaks the data:

• Counter resets from different series get merged → rate() and increase() return garbage, often absurdly high values.
• Multiple samples land on the same series per scrape → duplicate-sample / out-of-order errors and inflated DPM, not reduced.
• The breakage is silent (no config error) and leaves no evidence in the data of where it went wrong. Weeks later someone asks "why is my DPM so high / why is rate() absurd?" and there's nothing to point to.

The only safe remediations are:

1. Drop an entire unwanted metric (action: drop on __name__) — you're discarding the whole metric, not merging distinct series.
2. Fix the source — stop the application emitting the bad label (the real fix for unbounded path, user_id, etc.).
3. Adaptive Metrics — for structural cardinality on series you can't fix at the source. It aggregates correctly (counter-reset-aware, audited, reversible). This is the right way to reduce the cost of a label like pod. Route to adaptive-metrics.

Everywhere below that says "drop a label," read it through this rule: drop whole metrics, fix the source, or use Adaptive Metrics — never labeldrop a distinguishing label.

---

Symptom → Likely Cause

Symptom	Likely Cause	First Action
Prometheus OOMKilled or memory growing linearly	Active series growth (often from a new bad metric or label)	Active Series triage
Single PromQL query slow or OOMs the querier	One or more metrics in the query have high cardinality	Per-query drill-down
Remote write lagging, WAL growing	Sample throughput spike — series count OR scrape interval changed	Active Series triage + check scrape intervals
429 Too Many Samples / out of bounds errors	Hitting Mimir/Cortex ingester per-tenant series limit	Per-metric drill-down, find the new offender
Grafana Cloud Active Series bill spiked	New metric, new label, or rollout creating churn	Per-metric drill-down + churn check
Grafana Cloud DPM bill spiked but Active Series flat	Scrape interval shortened, OR remote_write sending duplicates	DPM-side issue — route to dpm-finder
series_limit_per_user errors after a deploy	Application change introduced a new bad label	Recent change diff
Series count grows then resets every restart	Series churn from ephemeral label values	Churn diagnosis