acl-experiments

ACL Experiments

Use this while the experimental story can still change. The ACL evidence bar is not "beats the baseline once": it is a defensible measurement of a language capability, with the failure modes examined.

Baseline honesty

• Include the strongest cheap baseline: a well-prompted current LLM has become mandatory context for most tasks — a method beating only pre-LLM systems invites the "does this matter now?" review.
• Tune baselines with the same care as your method (same search budget, same data); reviewers explicitly probe for asymmetric tuning.
• Report the trivial baselines (majority class, copy input, retrieval-only) when they contextualize how hard the task actually is.

Evaluation design

• Breadth must match the claim: a "general" claim needs multiple datasets; a cross-lingual claim needs typologically distinct languages, not three Romance neighbors.
• Automatic metrics need justification for generation tasks — pair n-gram or embedding metrics with human or LLM-judge evaluation, and validate any LLM-judge against human labels before leaning on it.
• Fix the evaluation protocol before final runs: dev-set peeking on the test set via repeated submissions is unreportable and unrepairable.

Statistical floor

The Responsible NLP checklist (Section C) asks for descriptive statistics and error bars — an experiment plan that cannot fill Section C truthfully is incomplete by construction.