multiple-choice-evaluation

Multiple-Choice Evaluation (LLM Judge, Not Regex)

The point

MCQ probes compute their headline metric (accuracy, P(chosen), Δaccuracy between arms, per-choice rate) by parsing a letter out of a free-form generation. Parsing is the measurement — a parser that mislabels 5–15 % of rows fabricates or destroys the Δ.

Naive case-insensitive letter regex has three symmetric failures:

1. False positive — letter inside a word. re.search(r"[AB]", ...) matches the first A/a/B/b anywhere: "At the garden…" → A, "a valid choice…" → A, "option B refers to…" before rejection → B. Fine-tuned / steered models that stop obeying "answer A or B only" produce exactly these strings.
2. False negative — over-tight regex. Tightening to \banswer[:\s]+([AB])\b misses "A.", "(A)", "I'd go with A", non-English preambles. The trade-off is inescapable inside a regex; disambiguation needs semantics.
3. Silent coercion. A two-bucket parser maps refusal / off-topic to whichever letter appears first (usually A). A 30 % refusal rate reads as a 30 % A-preference — a phantom effect that survives seeds because the coercion is deterministic.

Recipe

1. LLM judge on the full generation. Send the raw generation plus the question stem with all option texts (not only letters — text is what removes the ambiguity a regex can't resolve) to a capable judge; the final per-row verdict must be three-way CORRECT / INCORRECT / OTHER. Refusal / off-topic / multiple / none / format-broken all land in OTHER and are never coerced into CORRECT or INCORRECT; report separately or drop from the denominator. A strict regex is fine as a fast pre-filter, but any prose after the match falls back to the judge. Two prompt shapes work — pick one:

(a) Extract-then-compare. Judge only extracts which option the model picked (A / B / … / OTHER); a hard-coded step outside the LLM compares against the gold letter. Cleanly separates parsing from scoring, and the extracted letter is auditable on its own.