paper-verification

Paper Verification Methodology

You are helping a researcher verify that their paper accurately reflects their code and experimental results. This is the most critical quality control step in academic writing.

Verification Dimensions

1. Numerical Accuracy Audit

For every number in the paper (dataset sizes, metric values, percentages, counts):

1. Extract the number and its context from the .tex file
2. Trace it to its source: code output, result file, log, or tracking system
3. Verify the value matches exactly (watch for rounding, percentage vs decimal)
4. Flag any number that cannot be traced to a source

Template:

| Paper claim | Location (.tex) | Source file/code | Source value | Match? |
|-------------|-----------------|-----------------|-------------|--------|
| "13,999 frames" | abstract L3 | len(glob(labels/*.json)) | ? | ? |
| "4.2% improvement" | Table 2 | eval_results.json | ? | ? |

Common numerical errors:

• Rounding inconsistencies (3.14 in text, 3.1415 in table)
• Stale numbers from earlier experiments not updated after re-runs
• Percentage vs absolute confusion
• Off-by-one in dataset counts (headers counted, or not)

2. Terminology Consistency Audit

1. Extract all defined terms from the methods section
2. Search for each term across ALL sections
3. Flag any inconsistent usage:
   • Same concept, different names (e.g., "tag head" vs "classification head")
   • Same name, different meanings across sections
   • Defined but never used, or used but never defined

3. Code-Paper Alignment

For each method described in the paper:

1. Find the corresponding code (function, class, module)
2. Compare the paper's description with the actual implementation
3. Check specifically:
   • Algorithm steps match code flow
   • Hyperparameters in text match config/code defaults
   • Architecture descriptions match model code
   • Loss functions in equations match loss code
   • Training procedures match training scripts