panel

Panel — the verdict, and the evidence channel

The design problem here is not accuracy. It is that every artefact the verdict rests on is reachable by the thing being judged. Frontier coding agents modify tests, overwrite timers and monkey-patch evaluators to return success. On RE-Bench, a research-engineering benchmark, 30.4% of runs exhibited reward hacking, and on some tasks every successful run did (C14). An audit of a benchmark built specifically to be trustworthy found 59.4% of the audited subset materially flawed and retired it (C15).

So the sequence below snapshots the evidence before anything judges it, and the verdict is void if its digest does not match.

Running as a Gemini model? Read gemini.md in this directory first, then follow this file with the overrides it names. It puts denominators under the lane set, the captures and the verdict's five ledger fields, keeps relative qualifiers out of both the find and filter prompts, and reads model ids out of lanes.toml rather than from memory. Other models skip it.

Procedure

1. Snapshot first, judge second.

   # Resolve ONCE in the conductor and pass the absolute path down to every agent.
   # A spawned agent does not reliably inherit CLAUDE_PLUGIN_ROOT, so a relative
   # `scripts/...` path resolves against the agent's own cwd and dies.
   W=<abs path to warrant>/scripts
   python3 "$W/snapshot_evidence.py" --root <repo> --diff <patch> --tests <dir> --captures <dir>
   

   Prints a content-addressed digest over sorted paths and contents, writes the snapshot read-only, and that digest travels with every verdict. A verdict whose digest does not match its snapshot is void rather than suspect.

   This is the most-failed invocation in the toolkit, and the flags above are not the problem. Measured 2026-08-26: 19 of 46 real snapshot_evidence.py calls produced no snapshot, across 30 different agents, on two mistakes — an argparse usage: banner from an invented flag, and a script path that does not exist from the agent's working directory. Thirty agents meeting the same CLI for the first time made the same error, which is a documentation defect rather than thirty careless agents. The only flags that exist are --root --diff --tests --captures --include --json --now --selftest; there are no others, and --out, --dir and --evidence are the three most commonly invented.