auto-verify

Workflow 1.75: Claim Verification

Stress-test claims by swapping method, dataset, and model — applied uniformly to every claim regardless of whether the main experiment supported or rejected it. Each variant runs, gets judged against the main experiment's verdict on the frozen claim, and contributes to a per-claim robustness score; the claim passes if robustness ≥ ROBUSTNESS_THRESHOLD (default 0.5).

For: $ARGUMENTS

Overview

/auto-experiment produces per-run verdicts; /result-to-claim turns them into per-claim support. This skill asks a narrower but harder question: does the variant reach the same conclusion as the main experiment when we change the method, the dataset, or the model?

Per-variant we judge agreement as binary (pass / fail) — there is no middle "partial" tier; anything short of unambiguous agreement counts as fail. Per claim we aggregate as robustness = #pass / N_eligible, counted over the variants whose integrity check passed. The claim PASSes iff robustness ≥ ROBUSTNESS_THRESHOLD (default 0.5 — at least half of eligible variants must pass; e.g. at N=3 this means ≥2 of 3), otherwise FAILs — unless the eligible set is empty, in which case the claim is ZERO_ELIGIBLE_VARIANTS (a distinct terminal state, see below).

The pipeline runs in three stages, with two integrity gates:

1. Stage 1 — Setup & Gate (Phases 1–2). Parse arguments and identify target claims (no cap here — Stage 1 audits every target claim), then invoke two cross-model audits per claim on refine-logs/: /experiment-audit for evaluation methodology (fake GT, score normalization, phantom results, dead metric code, scope overclaim) and /mechanism-audit for mechanism-intervention rigor (steering coefficient sweep + reserved checks). The Phase 2 gate verdict for each claim is a single derived value combined = max_severity(exp.overall_verdict, mech.overall_verdict) with n/a (no mechanism intervention used) treated as pass. If combined = FAIL (i.e., at least one sub-audit returned FAIL → max_severity propagates FAIL), no PASS/FAIL on the claim is meaningful — the claim is marked INCONCLUSIVE here and Phase 3 step 0 short-circuits the rest of verify (Phases 3–10) to Phase 11 for that claim only. The inconclusive_reason names which sub-audit failed so iteration knows whether to fix evaluation, mechanism, or both.
2. Stage 2 — Run variants (Phases 3–7). Phase 3 step 0 picks the top-K admitted claims by importance (K = MAX_VERIFY_CLAIMS, default 1) — un-picked ADMITTED claims are marked INTEGRITY_ONLY with stage2_skip_reason: max_verify_claims_cap and skip to Phase 11. For each picked claim, pick alternatives, critique the plan, implement and (optionally) code-review each variant, deploy. Stage 2 ends when all variants have completed and raw metrics are on disk — no LLM judgment yet. Stage 2's exit gate lets you eyeball raw variant numbers before letting Stage 3 interpret them.
3. Stage 3 — Judge, Audit & Aggregate (Phases 8–11). For each variant of each picked claim, invoke /result-to-claim to judge whether its data supports the claim (Phase 8). Then run the variant-level integrity audit per claim (Phase 9) — symmetric to Phase 2, /experiment-audit + /mechanism-audit both run on the variant directory; each variant's integrity_status is the combined max_severity(exp, mech). Integrity-FAIL variants are excluded from both numerator AND denominator. Then compute robustness once on the post-audit eligible set and assign PASS / FAIL / ZERO_ELIGIBLE_VARIANTS (Phase 10). Finally write the report (Phase 11). If fewer than MIN_VARIANTS_FOR_VERDICT variants survive integrity (default MIN_VARIANTS_FOR_VERDICT=1, so only N_eligible = 0 triggers this), the claim is marked ZERO_ELIGIBLE_VARIANTS — variants did run, they just all failed integrity, so the iteration loop's instruction is "fix the variant evaluation, do not touch the main experiment."