release-management-and-rollback-planning

Release Management and Rollback Planning

The rollback plan is written before the release, not improvised during an incident. A release with no tested rollback path isn't a release plan — it's a bet that nothing will go wrong.

Progressive delivery

Ship changes to an increasing slice of traffic rather than all at once:

1. Canary: route a small percentage of traffic (often 1-5%) to the new version; compare error rate, latency, and business metrics against the baseline before continuing.
2. Staged rollout: widen in increments (e.g. 5% → 25% → 50% → 100%, or region by region / hotel-brand by hotel-brand), with a defined dwell time and defined go/no-go metrics at each step.
3. Full rollout: only after each prior stage cleared its metrics thresholds, not just "elapsed time with no complaints."

Config and data changes need the same discipline as code changes — a "just a config update" mindset is exactly what removes the staged rollout step that would have caught a bad config before it reached 100% of traffic (see Real-world grounding).

Feature flags

Feature flags decouple deploy (code reaches production, dormant) from release (the behavior is turned on for users), which is what makes canarying and instant rollback possible without a redeploy. Martin Fowler's feature toggle taxonomy usefully separates flags by purpose:

• Release toggles — hide incomplete work behind a flag so it can merge to main continuously; short-lived, removed once fully released.
• Ops toggles — a kill switch for degrading gracefully under load or disabling a risky feature instantly if it misbehaves; can be long-lived.
• Experiment toggles — A/B testing; lifespan matches the experiment.
• Permission toggles — gate a feature by plan/tenant/role; often long-lived by design.

Flag hygiene matters: a release toggle left in the code long after full rollout is tech debt (dead code paths, combinatorial testing burden) — schedule its removal at rollout time, not "eventually."