adversarial-empirical-review

Adversarial Empirical Review

Runs a regression-gated, N-round adversarial loop over a project's empirical output. The design is in docs/2026-06-08-adversarial-empirical-review-design.md (v2) and the module API in docs/CONTRACTS.md. Read the design before operating the skill.

The costly error here is corrupting a result that was already correct, so the whole pipeline is incumbent-preserving: a hard failure-set regression gate, "no clear difference keeps the incumbent" for residuals, and minimization of LLM judgment in favour of mechanical checks.

The crux is whether the tables are correct. Auditing an empirical manuscript means verifying that the numbers in its tables are what the analysis code actually computes — that each coefficient, standard error, p-value, N, and R² reproduces from the data and the code and is internally consistent. This is an empirical + code review: the battery, the reviewer, and the drafter are all scoped to table-number correctness, the manuscript is consulted only as the spine that maps paper Table numbers to fragment files, and prose is not examined (the prose-scanning catalog checks default to off; research-design and interpretation critique are out of scope, with the consequential-category machinery retained as a fail-safe). The mechanical anchor for this is builtin.table_source_agreement (every estout-fragment cell is audited against a numbers.json of computed values) together with the reproduction tier and the coef/SE/t, star/p, bounds, and sample-size checks. The crux is fail-closed: a manuscript table with no numbers.json spec is a FAIL and a missing artifact a CHECK_ERROR, and the driver will not converge unless every required_checks entry (default the crux) is present and passing — otherwise it stops with crux_unverified. A manuscript whose tables are hand-transcribed estout fragments decoupled from the regressions (the common case) is reviewed by recomputing the regressions and diffing the cells — supply a numbers.json of computed values so the audit is mechanical rather than left to the reviewer's eye.

Generation-based certification (the path past the parse-and-compare ceiling). Parse-and-compare has a structural limit: a second numeric token inside one cell, a number in a spanning descriptor row, or content behind an exotic macro is invisible to any LaTeX parser. The durable fix is to GENERATE the fragments from the computed source so displayed == computed by construction: scripts/tablegen.py renders estout-style fragments deterministically from layouts entries in the same numbers.json (the project's pipeline calls python tablegen.py --numbers ... --tables-dir ... after the analysis writes the artifact), and builtin.fragment_regeneration certifies each laid-out table by re-rendering it and comparing byte-for-byte with the on-disk fragment — any drift at all is a FAIL, so the parser gaps disappear for generated tables. The parse-and-compare crux keeps running on the same artifact as defense in depth. Mixed mode is supported (generated and hand-written tables side by side; the PASS message names the uncovered tables); require_all: true under battery.builtin["builtin.fragment_regeneration"] demands a layout for every manuscript table, and projects that adopt generation should add the check to required_checks so an unevaluated state blocks convergence exactly like the crux. Keys under tables starting with _ are auxiliary value rows (placeholder sources for composite cells), exempt from the crux's missing-row rule.

From-raw mode (snapshot_from_raw: true). The strongest re-execution tier: the driver copies the project, deletes every committed intermediate (snapshot-partition files, output-partition files, protected_paths caches, the numbers artifact — directory globs expanded, deletions confined to the copy), runs the full raw inputs → cleaning → analysis → tables pipeline (rebuild.from_raw_command, or rebuild.from_raw_argv as an exact argv list — preferred on Windows) twice, and certifies three checks: builtin.raw_table_identity (the regenerated LaTeX table files are IDENTICAL to the committed ones, modulo line endings/trailing whitespace), builtin.raw_numbers_agreement (numbers artifact within tolerance), and builtin.raw_determinism (the two runs agree). Activation is fail-closed: partitions.raw plus a from-raw command are required, the raw partition is read-only and provenance-hashed like the snapshot, the identity check is auto-added to required_checks so the run cannot converge while the from-raw chain is unverified, and driver init halts if any project file matches no partition glob (an unclassified file is a purge-coverage hole). The driver runs all of this inline — no orchestrator action is needed beyond stepping — but expect long battery phases: the full pipeline executes twice per re-execution, bounded by rebuild.from_raw_timeout (default 7200 s per run).