deployment-safety

Deployment Safety

A deploy is a bet that new code behaves in an environment you've never run it in. This skill makes the bet survivable: every deploy must be verifiable (you can tell if it worked) and reversible (you can undo it faster than users notice).

The two questions before any deploy

1. "How will I know within 5 minutes if this broke something?" — if the answer is "users will tell us", stop and add the signal first (health check, error rate, a smoke request you'll run).
2. "What is the undo?" — redeploy previous image/commit? feature flag off? config revert? If the undo involves a database, see the migration rules below, because code rolls back; data does not.

Gates

Gate 1: Build determinism

• The artifact deployed is the artifact tested: build once, promote the same image/bundle through environments. Rebuilding per-environment invites "works in staging" drift.
• Pin versions: base images by tag (ideally digest), lockfiles committed and honored (npm ci, pip install -r with pins). A deploy should not change behavior because a dependency released overnight.
• Know your build-time vs run-time config split. Anything baked at build (e.g. NEXT_PUBLIC_*) cannot be fixed by editing env vars later — document which is which.

Gate 2: Config & secrets parity

• Every env var the code reads exists in the target environment. Diff code's required config against the platform's actual config before deploying, not after the crash loop.
• Fail fast and loud on missing/invalid config at boot — a clear startup error beats a 3 a.m. NullPointerException.
• Verify the deployed commit: expose a version/commit fingerprint endpoint or log line so you can confirm what's actually running (never assume the deploy took).

Gate 3: Database migrations — the irreversible part

• Expand → migrate → contract. Never deploy a migration and the code that requires it as one atomic hope. Order: (1) additive migration (new nullable column/table) → (2) code that writes both/reads either → (3) backfill → (4) code that uses only new → (5) drop old, releases later.
• Destructive operations (DROP, ALTER TYPE, DELETE, renames) get: a stated rollback plan, a backup/snapshot taken first, and explicit user confirmation. A rename is a drop in disguise — do add+backfill+drop instead.
• Migrations must be safe to run twice (idempotent or guarded) and safe while old code is still running (the previous version and new schema coexist during rollout).
• Long locks kill production: on Postgres, add indexes CONCURRENTLY, avoid full-table rewrites in peak hours, set lock_timeout.