sre-canary

SRE Canary — Post-Deploy Monitoring & Canary Verification

Approach every post-deploy window as the engineer who will be paged if something breaks. A deployment is not done when code ships — it is done when you have confirmed that real production traffic is behaving the same as or better than before the change. Every minute of a degraded canary that you do not catch is a minute of user-visible failure that compounds.

The two failure modes are symmetric and equally dangerous: rolling back too early wastes a deploy and erodes team confidence; rolling forward on a broken canary causes real outages. This skill teaches you to read the signals precisely enough to tell them apart.

---

Step 0: Before You Start Monitoring

Before watching any signals, establish your baseline. Monitoring without a baseline is just staring at numbers.

1. Record the pre-deploy steady state — p50/p95/p99 latency, error rate, request rate, and Core Web Vitals from the last 30 minutes before the deploy.
2. Note the deploy timestamp exactly — you will overlay this on every graph. The most common source of confusion in post-deploy review is uncertainty about whether a signal shift happened before or after the change.
3. Identify the blast radius — which endpoints, services, user segments, or geographic regions does this deploy touch? Monitor those first; do not drown in unrelated signal.
4. Know your rollback command before you deploy — not after. The worst time to find your rollback runbook is during an active incident.
5. Define success criteria in advance — "error rate stays below 0.5%, p99 latency stays below 800ms, no new JS exceptions" is a success criterion. "Seems fine" is not.

---

Post-Deploy Monitoring Loop

Run this loop continuously during the canary window. Do not walk away after pressing deploy.

Phase 1: Immediate (0–5 minutes post-deploy)

The first five minutes catch hard failures: startup crashes, broken health checks, misconfigured routing, missing environment variables, and database migration errors.