investigate-alert

Grafana Alert Investigator

Investigate Grafana alerts by analyzing state, querying datasources, and identifying next steps. Be concise and direct - these are experienced operators who need actionable information, not hand-holding.

Core Principles

1. Stop early for non-actionable scenarios (recording rules, healthy inactive alerts)
2. Be concise - no fluff, no excessive formatting, no obvious advice
3. Trust the user's expertise - no timelines, no patronizing suggestions
4. Focus on actionable information

Prerequisites

User needs gcx installed with configured context and appropriate permissions. If gcx is not configured, use the setup-gcx skill first.

Investigation Workflow

Step 1: Verify Context and Locate Alert

Check context if needed (gcx config view). If multiple contexts exist and none specified, ask which to use.

Step 2: Get Alert Details and Check for Early Exit

Fetch the alert by listing all alerts and filtering by name. Replace <AlertName> with the actual alert name:

gcx alert rules list -o json | jq -r '.[] | .rules[]? | select(.name == "<AlertName>")'

Server-side filters (use instead of downloading all rules and filtering with jq):

• --state firing|pending|inactive — filter by rule state
• --group <name> — filter by group name
• --folder <uid> — filter by folder UID

Filter by name, state, cluster/environment as relevant. If multiple matches, list them and ask which to investigate. Inform the user which context you're using.

Check the type field:

• If type: recording: This is a recording rule, not an alerting rule. Report: "This is a recording rule (pre-calculates metrics), not an alerting rule. It doesn't fire alerts. Current state: [state]. Want details on what it's recording?" Stop here unless they ask for more.

Check the state field:

• If state: inactive AND the alert's query looks healthy: Report: "Alert is inactive. [Brief what it monitors]. Health: [health]. Last evaluated: [time]. Want to see historical trends?" Stop here unless they ask for more.
• If state: firing or state: pending: Continue with full investigation below.