investigating-alerts

Alert investigation priority

For questions about alerts, incidents, or "what's wrong":

1. List available datasources to discover their UIDs — reuse these UIDs for the rest of the session
2. Check Prometheus datasource alerts first (pass the Prometheus datasource UID); filter by label_selectors with the alert's alertname label — search_rule_name is ignored on the datasource path and returns all rules (a large token cost)
3. Check Grafana-managed alerts (without a datasource UID filter)
4. Cross-reference with logs, traces, and metrics for context

Why Prometheus first? Most alerting rules live in Prometheus datasources, not Grafana-managed alerts.

Root cause analysis workflow

1. Gather evidence — Query alerts, logs, traces, and metrics in parallel
2. Find correlations — Look for timing patterns across data sources
3. Narrow down — Use specific label filters once you identify the affected component
4. Verify — Confirm the root cause with targeted queries before proposing solutions

Investigation discipline (this request)

This turn is an alert investigation. Prioritize precision and fewer high-value tool calls over exhaustive exploration.

• Runbook ordering — The user prompt requires checking the runbook_url annotation before deep investigation. Treat that as binding: fetch and apply the runbook before broad discovery.
• Anchor on the alert — Use the alert name, labels (namespace, cluster, service, job, severity), and any text in the notification to choose narrow filters. Do not run cluster-wide label enumeration when the alert already identifies a scope.
• Tight parallel batches — Parallel tool calls should share the same incident time window and suspected blast radius (e.g., alert row + metrics for the labeled job + logs for that service). Avoid parallel calls that scatter across unrelated systems without a hypothesis.
• Sufficiency — When metrics or logs support a likely root cause and you can name a single verification step, conclude. Do not continue investigating every datasource for completeness.
• Final answer shape — Lead with a short verdict (most likely cause), then evidence (queries, samples), then remediation and follow-up checks.