ML-MLOps-Portfolio: Skill for Claude Code

Performance Degradation RCA — Multi-stream root cause

This skill is the post-incident counterpart to /rollback. Rollback restores service; this skill explains WHY degradation happened and what long-term fix is needed to prevent recurrence.

When NOT to use this skill

• Single-slice performance alert — use concept-drift-analysis first; it's narrower and faster.
• Active incident with user impact — run /rollback first, this skill after the dust settles.
• Drift-only without performance impact — use drift-detection; degradation RCA assumes ground truth has already revealed regression.

Evidence streams

The skill correlates FIVE independent streams, then synthesizes:

1. Sliced performance metrics (from performance_monitor Prometheus push) — which slices degraded, by how much, when
2. PSI drift history (ops/drift_reports/*.json) — was the distribution shift gradual or sudden?
3. Deploy + release history (MLflow Registry, kubectl rollout history, GitHub Releases) — did a deploy precede the regression?
4. Upstream data (feature store / ETL provenance) — did an upstream schema or semantic change leak through?
5. Prediction log statistics — score distribution shift, error rate per version, logger error rate (if this spiked, our verdict is uncertain — flag it)

Execution flow

Step 1 — Incident intake (AUTO, 2 min)

# Confirm the incident is declared
gh issue view {incident-id} --json labels,body,createdAt

# Pull the current Prometheus alert state
curl -s "http://prometheus.monitoring:9090/api/v1/query?query=ALERTS{service='{service}'}" | jq '.'

# What alert(s) kicked this off? Record them as the starting hypothesis.

Output: a single-paragraph framing — "At {ts}, {alertname} fired for {service} claiming {metric}={value}. Impact window: {start}→{end}."

Step 2 — Sliced regression scan (AUTO, 3 min)

# What slices are below baseline? Which are ABOVE?
curl -s 'http://prometheus.monitoring:9090/api/v1/query_range?query=\
{service}_performance_metric{metric="auc"}&start=...&end=...&step=1h' | \
  jq '.data.result' > ops/incidents/{incident}/sliced_auc.json

# Rank slices by magnitude of regression (current - baseline)
python scripts/rank_regressions.py ops/incidents/{incident}/sliced_auc.json