reviewing-paper-sources

Reviewing Paper Sources

Overview

A paper review is an evidence-gathering exercise, not a reading exercise. The findings that change a paper come from things you compute or verify, not from things you notice while reading: recomputing the headline statistic, checking every reference against its publisher record, compiling the sources yourself, rendering the figures at print size.

Core principle: never trust an artifact you did not produce. The committed PDF is stale until proven current. A reference is wrong until checked against a primary source. A reported rate means nothing until you have its sample size.

Mode: establish this first

	Mode A — internal	Mode B — external
Situation	Advisor/self-review of your own group's draft	Referee report for a venue
Register	Blunt, shared stake in the outcome	Neutral, addressed to authors + chair
Proposes fixes	Yes, including edits to the sources	No — diagnose only, never rewrite the authors' text
Artifacts	Review .md + .pdf, *-annotated.tex, refs-corrected.bib	Review .md + .pdf only
Scores/verdict	Yes (both modes)	Yes

If it is ambiguous, ask. "My student's paper" ⇒ Mode A. "I'm reviewing for X" ⇒ Mode B.

Then establish what you were given. With LaTeX sources, follow the workflow below as written. With only a PDF — the usual case for refereeing — the compile step, the .bbl cross-check and the source-based tools are all unavailable, and two failure modes appear that do not exist otherwise. Build the pseudo-source first and read reference/pdf-only-review.md:

python3 assets/run-pdfshim.py paper.pdf --out review-assets/

Also establish where in its lifecycle the paper is, because it changes what the review is for: pre-submission (triage by what fits the deadline), already submitted (target what reviewers will attack, and prepare answers), or post-rejection (framing and structural weaknesses).

An unfinished draft is a fourth case, and it needs the findings split three ways: content, not-yet-written, and container. Content is the only one that bears on the judgement — claims that are written and wrong, framing the paper's own data refutes, work missing from the argument. Container is everything that makes the artifact submittable rather than good, and it belongs in its own late section with the severity tag saying so ("Blocks submission; not a judgement on the work") rather than sitting at the top of the concerns list outranking the science. The temptation runs the other way, because container defects are the easiest to find and the easiest to state with certainty — which is exactly why they crowd out the findings that took real work to establish.