model-migration-plan

Model Migration Plan Skill

A model swap changes every output of your feature at once. This skill plans the migration like the risky deploy it is: eval first, shadow second, canary third — with numbers, not vibes, deciding each promotion.

What This Skill Produces

• A phased migration plan (eval → shadow → canary → full) with promotion criteria per phase
• Prompt adaptation notes — what typically shifts between models and what to re-tune
• Rollback triggers and the mechanics of rolling back fast
• A cost/latency delta forecast for the new model

Required Inputs

Ask for (if not already provided):

• Current and target model (and why: deprecation, quality, cost, latency)
• The feature's traffic and blast radius — requests/day, who sees the output, what a bad output costs
• Existing evals — a regression suite (see prompt-regression-suite) or at minimum golden examples; if none exist, phase 0 is building one
• The deadline, if the migration is forced by a deprecation date

Migration Phases

Phase 0 — Baseline. Freeze a regression suite against the current model. Without a baseline, "the new model is fine" is unfalsifiable. Record current cost, latency (p50/p95), and quality scores.

Phase 1 — Offline eval. Run the suite against the target model with the prompt as-is, then with adapted prompts. Promotion criteria: pass rate ≥ baseline, no canary failures, cost/latency within budget. Expect to iterate here — most "model regressions" are prompt-fit issues.

Phase 2 — Shadow. Mirror a sample of real traffic to the new model; log, never serve. Compare distributions: refusal rate, output length, format-violation rate, judge scores on a sample. Duration: long enough to cover weekly traffic patterns.

Phase 3 — Canary. Serve the new model to [1-5]% of traffic behind a flag, tagged in analytics. Watch the same metrics plus user-visible signals (regenerate rate, thumbs-down, support tickets). Widen in steps; each step has the same promotion criteria.