paper-audit

Paper Audit

Claims in the paper must match code in the repository.

This skill audits research papers against their publicly available codebases to detect mismatches, missing implementations, and reproducibility issues. Inspired by Feynman's /audit workflow.

"Trust but verify." — Russian proverb

---

1. When to Audit

Audit when:

• Reviewing a paper with an accompanying GitHub repository
• Planning to build on someone's method (verify it works first)
• Reviewing for a conference/journal (check reproducibility)
• Teaching (show students what to look for)

Do NOT audit when:

• No code is available (audit becomes impossible)
• Paper is purely theoretical (no implementation to check)
• You are the author (use internal-critique instead)

---

2. The Audit Protocol

Step 1 — Paper Claim Extraction

Extract all verifiable claims from the paper:

Claim ID	Section	Claim	Verifiable?
C1	Method	"We use Adam optimizer with lr=0.001"	Yes
C2	Method	"Batch size is 256"	Yes
C3	Results	"Achieves 95.2% accuracy on ImageNet"	Yes
C4	Method	"Our attention mechanism reduces complexity to O(n)"	Partial (need to read code)
C5	Ablations	"Removing component X drops performance by 3%"	Yes

Claim types:

• Hyperparameter: Learning rate, batch size, weight decay, architecture details
• Architecture: Layer counts, dimensions, activation functions, connectivity
• Data: Dataset version, preprocessing steps, augmentation pipeline
• Metric: Reported numbers with confidence intervals
• Ablations: Performance differences when components are removed

Step 2 — Code Mapping

For each verifiable claim, locate the corresponding code:

Paper Claim → Code Location → Verification Result

Example:
C1: "Adam optimizer, lr=0.001"
  → Code: train.py:45: optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
  → Result: ✅ MATCH

C2: "Batch size is 256"
  → Code: config.yaml: batch_size: 128
  → Result: ❌ MISMATCH (paper says 256, code uses 128)

C3: "95.2% accuracy on ImageNet"
  → Code: README.md reports 94.8%
  → Result: ⚠️ DISCREPANCY (0.4% gap, possibly different eval protocol)