slo-investigate

SLO Investigator

Deep-dive investigation of breaching SLOs: dimensional breakdown, alert correlation, runbook access. For experienced operators — no hand-holding.

Core Principles

1. Use gcx commands — do not call Grafana APIs directly (no curl, no HTTP libraries)
2. Trust the user's expertise — skip obvious context, get to the root cause
3. Use -o json for agent processing, default format for user display; show graphs for time-series data
4. Errors collected at the end — do not interleave error handling in workflow steps
5. Use --from/--to for all time-range commands (never --start/--end)

Investigation Workflow

Step 1: Retrieve SLO Definition

gcx slo definitions get <UUID> -o json

Extract from the JSON response:

• .metadata.name — SLO name
• .spec.query.type — query type: ratio, freeform, or threshold
• For ratio: .spec.query.ratio.successMetric, .spec.query.ratio.totalMetric, .spec.query.ratio.groupByLabels[]
• For freeform: .spec.query.freeform.query
• .spec.objectives[0].value — objective (0–1), .spec.objectives[0].window — window
• .spec.destinationDatasource.uid — Prometheus datasource UID
• .spec.alerting.fastBurn.annotations, .spec.alerting.slowBurn.annotations — runbook/dashboard URLs
• .metadata.annotations — additional runbook/dashboard references

If no UUID is given, list SLOs and ask which to investigate:

gcx slo definitions list

Step 2: Check Status with Wide Output

gcx slo definitions status <UUID> -o wide

This shows SLI, ERROR_BUDGET, BURN_RATE, SLI_1H, SLI_1D, and STATUS.

Early exit — OK status: If STATUS is OK, report health metrics and stop:

SLO: <name> — Status: OK
SLI: <value>  |  Error budget remaining: <budget>%  |  Burn rate: <rate>x
1h SLI: <sli_1h>  |  1d SLI: <sli_1d>
No action needed.

Early exit — NODATA status: If STATUS is NODATA, branch to NODATA diagnosis:

SLO: <name> — Status: NODATA
Recording rule metrics unavailable. Likely causes:
- Destination datasource misconfigured (check .spec.destinationDatasource.uid)
- Grafana recording rules not yet evaluated (can take 1–2 minutes after creation)
- Prometheus federation/remote write issue

Check: gcx datasources list --type prometheus
Then verify the destination datasource UID matches what the SLO expects.