rollback

Rollback — Emergency Procedure

This is the MOST CRITICAL skill in on-call. It must be predictable, idempotent, and auditable. Execution is STOP-class: the agent produces the plan; a human approves each destructive command.

When NOT to use this skill

• Flaky alert without user impact → investigate with concept-drift-analysis first. Rollback has its own blast radius.
• Degraded single slice, global healthy → targeted retrain, not rollback.
• Canary already auto-aborted by Argo Rollouts → no action needed; verify and document.
• Pre-canary issues (CI failures) → use the deploy skill's abort path, not this one.

Prerequisites

Before running the plan, verify:

• kubectl config current-context points at the affected cluster
• kubectl argo rollouts version is available OR kubectl with the rollout subcommand for plain Deployments
• GitHub token has repo + actions scopes for issue creation
• MLflow Tracking URI is reachable (check via mlflow experiments list)

Decision tree

              Incident declared
                      │
         ┌────────────┴────────────┐
     Argo Rollouts?          Plain Deployment?
         │                         │
  kubectl argo rollouts        kubectl rollout undo
  abort + undo                 deployment/{service}
         │                         │
         └────────────┬────────────┘
                      │
          Wait for Ready (kubectl rollout status)
                      │
         ┌────────────┴────────────┐
     Model changed?            Image-only change?
     (MLflow revert)           (done)
         │
  MLflow transition_model_version_stage
  Previous → Production
         │
  POST /model/reload to each pod (or restart)
         │
  Silence incident-related alerts
         │
  Open audit issue (labels: rollback, incident)

Step 1 — Confirm the incident (AUTO, 1 min)

# Who is firing?
kubectl get prometheusrules -n monitoring -o json | \
  jq '.items[].spec.groups[].rules[] | select(.alert != null) | .alert'

# Current rollout state
kubectl argo rollouts get rollout {service}-predictor -n {namespace}

# Recent deploys
kubectl rollout history deployment/{service}-predictor -n {namespace}
kubectl argo rollouts history {service}-predictor -n {namespace}