regression-evals

Regression Evals

Every prompt change, model upgrade, or tool tweak is a potential regression. Regression evals catch breakage before users do — if you gate deploys on them.

When to Use

• Any AI feature in production
• Before upgrading model versions
• Before merging prompt changes
• Weekly as a drift/decay check

The Core Loop

1. Maintain a versioned eval dataset (see eval-dataset-design)
2. On every proposed change, run evals on baseline AND candidate
3. Compare per-stratum metrics with significance tests
4. Gate merge on: no stratum regresses beyond threshold

Minimal Harness

interface EvalCase { id: string; input: any; expected: any; stratum: string }

async function runEvals(model: string, prompt: string, cases: EvalCase[]) {
  const results = [];
  for (const c of cases) {
    const output = await runModel(model, prompt, c.input);
    const score = await scoreOutput(c.expected, output); // 0-1
    results.push({ id: c.id, stratum: c.stratum, score });
  }
  return results;
}

const baseline = await runEvals("claude-sonnet-4-6", oldPrompt, cases);
const candidate = await runEvals("claude-sonnet-4-6", newPrompt, cases);

const regressed = compareByStratum(baseline, candidate, { alpha: 0.05 });
if (regressed.length) throw new Error(`Regressions in: ${regressed.join(", ")}`);

Significance Testing

A 2% drop on 100 items is noise, not signal. Use bootstrap confidence intervals:

import numpy as np
def bootstrap_ci(scores, n=1000, alpha=0.05):
    means = [np.mean(np.random.choice(scores, size=len(scores), replace=True)) for _ in range(n)]
    return np.percentile(means, [100*alpha/2, 100*(1-alpha/2)])

baseline_ci = bootstrap_ci(baseline_scores)
candidate_ci = bootstrap_ci(candidate_scores)

# If CIs don't overlap AND candidate lower, it's a real regression.

For pass/fail metrics, use McNemar's test on the paired outcomes. For scalars, paired bootstrap.

Thresholds

Hard rules:

Metric	Regression threshold	Action
Any stratum	≥ 3% drop with p < 0.05	Block merge
Aggregate	≥ 5% drop with p < 0.05	Block merge
Single catastrophic item	New case drops from pass→fail	Investigate, likely block
Variance	CI widens significantly	Investigate (noisier outputs)