gauntlet

Gauntlet

A skill is a claim: "the model does better with me loaded." Nobody gets to assert that — they prove it. Reading a skill tells you what it intends; only a controlled run tells you what it does. Gauntlet found a polished skill scoring WORSE than no skill at all, and found the mechanism (its evidence pressure manufactured fabrications). That class of defect is invisible to review and fatal in use.

The Law

1. Control arm or it didn't happen. Every experiment includes a no-skill arm on the same task. A skill is only good if it beats the model without it.
2. Blind judging, shuffled labels. The judge never knows which output used a skill, which version, or that skills are involved. Shuffle label order between rounds so position bias can't repeat.
3. The skill under test never sees the rubric. Fixtures and ground truth live outside anything the tested arm reads. And the fixture must be SYNTHETIC — arms must not be able to read the real subject system. codex exec --sandbox read-only restricts writes, not reads; -C <dir> sets cwd and jails nothing; claude -p with bypassPermissions reads the whole machine. So a fixture naming a real repo, service, standard, or ticket that exists on this disk is not a fixture: arms silently cross-check it against reality, and the judge then scores TRUE findings as fabrications because the rubric assumed they could not know. Verified 2026-08-12 — a fake PR about a real service produced arms citing the real requestWithRetry.go:147 and the real docs-repo HEAD SHA, and a 33-count "fabrication" score that was mostly correct reads. Invent service names. Verify containment empirically by grepping every arm's output for real paths and 40-char SHAs BEFORE judging, and discard any arm that escaped. Capabilities must also be matched across arms: one arm with network, gh, or MCP access and another sandboxed are not the same experiment, and the fabrication axis becomes garbage.
4. Single runs give direction, not decimals. Model output varies run to run; treat a 0.5 gap as noise, a 3-point gap as signal. Rerun ALL arms fresh when comparing versions — never compare a fresh run to a cached one.
5. Findings become proposals, not automatic rules. Recommend the smallest supported correction; deletion, simplification or no change may be right. Evaluation does not authorize changing an installed skill or committing an artifact. Show the exact proposed diff and follow the global approval gate before adoption; then rerun the relevant controlled comparison.
6. Detection saturates. Frontier models catch planted flaws easily; when every arm scores full marks, the fixture is dead — the differentiators become fabrication, calibration, presentability, and pressure stability. Refresh fixtures with subtler flaws rather than celebrating ties.