eval-design-forensics

Eval-Design Forensics — does the evaluation measure what the paper claims?

Audit evaluation-design and reporting validity for: $ARGUMENTS (requires claims.json from /evidence-ledger). Emit span-anchored eval-design-forensics.findings.json. This skill computes no verdict.

🔒 Run once per input change; never wrap in /loop, /schedule or CronCreate (${CLAUDE_PLUGIN_ROOT}/support/references/run-cadence.md). Re-run only when the paper or the ledger changes, or a repo arrives and raises the level to L2.

Adapted from the ML-evaluation-methodology literature — the leakage taxonomy of Kapoor & Narayanan (2023), the LLM-as-judge validity work (MT-Bench self-enhancement, self-preference, position bias), and the "Show Your Work" / reproducibility-checklist reporting norms — reframed to audit a third party's evaluation. A favourite autoresearch shortcut is to report a number that is arithmetically self-consistent (family A), runs real code against a real ground truth (family D), and still does not measure what it claims: the protocol leaks, the load-bearing metric is a conflicted/unvalidated LLM judge, or the reporting quietly drops a declared condition. This skill is the constraint that asks "is this a valid measurement of the claim?", pointed at a submission, and it stays honest — leakage and under-reporting are usually honest methodological errors, so every finding is a discrepancy to clarify, never an accusation.

Read reference/rationale.md for why this skill exists (the leakage / judge-validity / selective-reporting failure modes it targets).

Core principle

Ledger-anchored, span-verified, reviewer≠adjudicator, honest about what it cannot settle. Four properties:

1. Anchor to a PAPER claim. Every above-info finding cites a ledger claim_id and quotes a verbatim span of that claim's text_span (references/integrity-forensics-contract.md rules 1–2). The leak/judge/reporting tell lives in the protocol / setup-description — usually method and scope claims, with comparison / number for the judge metric and caption / table_cell / baseline for reporting. The anchor is whichever paper claim the finding undermines; a split-file file:line, a config, or a leaderboard date is forensic context for the description, never the anchor.
2. The executor assembles facts; the reviewer judges. At L2 the executor gathers mechanical split/preprocessing/judge/result facts (grep/hash — listing what exists is a fact, not a judgment) and may record one public-record date fact (a benchmark's release vs a model's cutoff, for the contamination FP guard). It passes paths + the ledger + those facts + the checklist to the reviewer and never pre-declares "this leaks" (references/reviewer-independence.md). The model proposes; tools/adjudicate_findings.py decides. This skill computes no verdict.
3. Undecidable leakage subtypes → hand off, don't guess. An illegitimate-proxy feature, sampling bias in the test set, and pretraining/benchmark contamination are domain / black-box judgments not settleable from the PDF or the repo. Emit verdict_local: needs_external_check + requires_external_check: true (contract rule 6); name the external methods a domain check would use — exchangeability (Oren 2023), Min-K% Prob (Shi 2023), Time-Travel (Golchin 2023), BIG-bench canary strings — and never run them.
4. Verdict-bearing at L0/L1; observability still caps the L2-confirm. Unlike experiment-forensics (no eval code at L0/L1 ⇒ info-only), a stated-tell here is decided from the described protocol and emits observability_level_required: 0. The L2 confirmation of the same leak/ omission is a separate finding with observability_level_required: 2 that auto-demotes on a PDF-only run (references/observability-levels.md). So a PDF-only run keeps the stated-tell as a flag and the verification as an info "confirm-at-L2" pointer — never the reverse.