llm-campaign-drift-gate

LLM Campaign Drift Gate

Problem

Batch-scoring campaigns (exposure measures, classifiers, extraction runs) call provider aliases that can be silently repointed to a new model at any time. Resuming a half-finished campaign after the alias moves splices two different scorers into one variable, with the version boundary correlated with whatever orders the chunks (time, firm id) — a silent confound. Providers can also RETIRE the old model entirely, making the original campaign uncompletable.

Context / Trigger Conditions

• Resuming a scoring run more than ~a day after its last paid chunk
• "Top up credits and finish the run" requests
• Any incremental scoring against an existing response cache
• Symptom of a missed gate: a step-change in scores at a resume boundary

Solution

Before ANY production spend on resume, run a two-part gate (~$0.30–2):

1. Canary (the decisive check): sample ~100 already-cached items, re-send their EXACT stored prompts fresh, compare fresh vs cached scores. Gate: ≥97% all-field exact match and no systematic directional shift. Write the comparison in a standalone script — never through the pipeline's cache layer, which would overwrite production entries.
2. Gold re-validation: re-score the gold/validation panel fresh and compare agreement metrics to the prior validation (e.g. median F1/κ within ~0.03, no domain dropping >0.10).

Also capture response.model on every gate call — pipelines rarely store it, and it is the only direct evidence of a repoint. Check the provider's /models endpoint: if the old model id is gone, no rollback exists.

1. If the canary fails, diagnose BEFORE concluding — two mandatory follow-ups:
   • Date the suspected flip against the provider's changelog before inferring a model splice. response.model on fresh calls identifies today's model only; if the alias already pointed there when the cache was written, there is no family splice and the mismatch needs another explanation. (Verified failure mode: an alias that had served the "new" model for months was misread as a fresh repoint.)
   • Fresh-vs-fresh canary to separate serving drift from temperature-0 nondeterminism: re-score the same items a second time. Drift signature = fresh2-vs-fresh1 agreement high and symmetric while both fresh runs disagree with the cache at a higher rate in the SAME signed direction. Noise signature = fresh-vs-fresh disagrees about as much as fresh-vs-cache, with no directional bias.
   • Supporting forensic: compare raw-response formatting fingerprints (JSON pretty/compact ratio, key order) between cache and fresh — a heterogeneous or shifted style distribution corroborates a serving change when no model id was recorded.