aaai-reproducibility

AAAI Reproducibility

Use this when a draft needs to survive AAAI review on rigor, not just novelty. AAAI-27 requires the reproducibility checklist to be uploaded separately from the main PDF, in its own field on the submission form (AAAI-26 carried it inside the PDF after the references) — so it is a document a reviewer opens on its own, and it has to agree with the paper and supplement rather than read as an afterthought. AAAI-27 also states that reviewers assess reproducibility from what was actually submitted, and that material promised "after acceptance or publication" is not evidence it exists.

Reproducibility audit

• Map each central claim to submitted evidence: theorem, table, figure, ablation, appendix item, checklist answer, or code/data artifact.
• Record seeds, splits, preprocessing, hyperparameters, model selection, early stopping, prompt selection, and hardware.
• Report variance or uncertainty when stochasticity affects conclusions.
• Document dataset licenses, access constraints, sensitive data, human-subjects issues, and annotation procedures.
• Separate training compute, inference compute, and experiment search cost.
• Check the reproducibility checklist for contradictions with the main text and supplement.

Common AAAI weaknesses

• Checklist says code/data are available but supplement lacks runnable commands.
• Main results rely on one seed, one benchmark, or one prompt family.
• Baselines are weaker than current open-source or widely cited systems.
• Evaluation uses closed data or APIs with no reproducibility substitute.
• Human evaluation omits annotator instructions or quality control.

Checklist-to-evidence consistency grid

AAAI places the reproducibility checklist after the references, and reviewers cross-check each "yes" against the paper and supplement. A "yes" with no backing artifact reads worse than an honest "no", because it signals the checklist was filled in carelessly.