experiment-forensics

Experiment Forensics — are the reported results what the code computes?

Audit experiment integrity for: $ARGUMENTS (a paper-dir or repo-dir; use an ABSOLUTE path — it is referred to as TARGET below). Emit span-anchored experiment-forensics.findings.json.

🔒 Run once per input change; never wrap in /loop, /schedule or CronCreate (${CLAUDE_PLUGIN_ROOT}/support/references/run-cadence.md). Re-run only when the observability level rises (a repo or result files arrive → L2).

Read reference/rationale.md for the ARIS lineage, the ten failure modes this skill targets (why it exists) and the two independence axes (executor ≠ reviewer, reviewer ≠ adjudicator); it is background, not procedure.

How this differs from the other auditors (route correctly)

This skill asks one question: are the reported numbers what the eval code computes (L2)? Text-only contradiction and text scope go to consistency-audit; "first / SOTA" external truth to baseline-comparison-audit + citation-forensics (needs_external_check); evaluation-design validity (leakage, LLM-as-judge) to eval-design-forensics; surface tells to presentation-signals. An LLM that produces ground-truth labels stays here as HP-FAKE-GT. Read reference/routing.md for the full auditor table and hand-off list when a finding might belong elsewhere.

Pipeline role + the anchoring model (read before running)

This is an auditor skill in the integrity-forensics pipeline (references/integrity-forensics-contract.md):

/evidence-ledger  →  claims.json (+ artifact_manifest.json, observability level L)
                          │
       experiment-forensics  ── reads the ledger, PROPOSES findings ──►  experiment-forensics.findings.json
                          │
   tools/adjudicate_findings.py  (deterministic; the ONLY thing that computes a verdict)

• It reads the ledger, it never re-reads the raw paper to invent structure, and it never computes the overall verdict — the orchestrator runs tools/adjudicate_findings.py for that. This skill stops at emitting findings.
• The anchor is always a PAPER claim. Every above-info finding cites a ledger claim_id and quotes a verbatim span of that claim's text_span (the paper number/scope/method sentence it undermines). The eval-code smoking gun (src/eval.py:88) is not a ledger claim, so it lives in the finding's description / recommended_reviewer_action, never as the anchor. No paper claim to anchor to ⇒ the finding cannot rise above info. (See worked examples.)
• Observability caps severity. Findings declare observability_level_required. Every code/result-level pattern is decidable only at L2; at L0/L1 it is emitted as info (Step 2). tools/adjudicate_findings.py is the structural backstop — any observability_level_required above the run's level is demoted to info.