investigating-alerts

Alert investigation priority

For questions about alerts, incidents, or "what's wrong":

1. If an Alert Context block was prefetched into the system prompt, skip rule discovery entirely and start from its metrics/matchers/runbook
2. Otherwise, list available datasources to discover their UIDs — reuse these UIDs for the rest of the session
3. Check Prometheus datasource alerts first (pass the Prometheus datasource UID); filter by rule title or label selectors — on mcp-grafana < 1.4.0 search_rule_name is ignored on the datasource path and returns all rules (a large token cost)
4. Check Grafana-managed alerts (without a datasource UID filter)
5. Cross-reference with logs, traces, and metrics for context

Why Prometheus first? Most alerting rules live in Prometheus datasources, not Grafana-managed alerts.

Root cause analysis workflow

1. Triage — Before deep queries, rank candidates using cheap signals: which alert fired first, which fires most often or carries the highest volume, and which signals (metrics + logs + traces) agree on the same component. Multi-modal agreement beats any single loud signal.
2. Localize with metrics — Metrics answer which component and which resource: use the alert's exact metrics and label matchers first (the Alert Context block when present)
3. Identify fault type with logs — Logs answer what kind of failure (OOM, connection refused, 5xx spike, backpressure): search error logs for the component metrics implicated
4. Confirm the path with traces — Traces answer how the failure propagates: use them only to confirm a specific propagation path you already suspect; do not mine raw traces hunting for a cause
5. Narrow down — Use specific label filters once you identify the affected component
6. Verify — Confirm the root cause with targeted queries before proposing solutions

Hypothesis discipline

• Keep 2–3 competing hypotheses alive early. Each names a component AND a fault type, plus the one piece of evidence that would falsify it. Do not anchor on the first plausible story (the most common RCA failure mode) and do not chase a single hypothesis with ever-similar queries.
• Check time ordering — A plausible cause must start BEFORE its symptoms. Compare first-seen times across metrics and logs; a "cause" that starts after the effect is a symptom, not a cause.
• Check topology consistency — When the system prompt includes a Service Topology block, a propagation path must follow those edges. Do not invent dependencies that are not listed.
• Self-critique before concluding — Before the final answer, ask: what is the strongest piece of evidence AGAINST my leading hypothesis, and did I look at it? If you have not, spend one query on it instead of a fourth confirming query.