aistats-experiments

AISTATS Experiments

Use this before submission when the empirical or simulation story is not yet locked.

Experiment audit

• Map each empirical claim to a table, figure, simulation, ablation, or robustness check.
• Include baselines that represent both ML practice and relevant statistical methods.
• Separate synthetic simulations that validate assumptions from real-data experiments that show practical relevance.
• Report uncertainty for stochastic results: repeated runs, standard errors, confidence intervals, paired tests, or bootstrap intervals when appropriate.
• Report dataset splits, preprocessing, metrics, hyperparameter search ranges, final chosen settings, selection criteria, random seeds, hardware, software versions, and runtime.
• Add ablations for the mechanism, not just cosmetic variants.
• Audit for leakage, selection bias, multiple-comparison issues, and mismatch between theoretical assumptions and empirical setup.

What experiments are for at this venue

• AISTATS experiments exist to validate theory, not to win leaderboards. One focused simulation confirming a predicted rate outweighs five extra benchmark datasets.
• The strongest design triad: a synthetic study where assumptions hold exactly, a study where they are deliberately violated, and a real-data study showing practical behavior.
• Reviewers, frequently statisticians, check whether the empirical regime — sample size, dimension, noise level — matches the asymptotic regime of the theorems. A bound proven as n grows but tested only at n = 500 invites the question of relevance.

Theory-validation design table