pre-analysis-plan

Pre-Analysis Plan

Overview

If you can run a hundred reasonable specifications and report the one you like, you haven't measured an effect — you've measured your own preferences with extra steps. A pre-analysis plan (PAP) is the commitment that stops this: write down what you will do, and what would count as the answer, before the outcomes are visible. It's the analytics counterpart of writing a spec before code — decide the contract first — but the stakes are higher: the temptation to retrofit the plan to the result is enormous and almost invisible after the fact.

Core principle: A specification chosen after seeing the outcome is not evidence. Decide the analysis while you are still ignorant of how it will turn out.

When you actually need this

Not every exploration needs a PAP — forcing one onto genuine EDA is theater (see question-framing and the exploration regime in data-contracts). It earns its weight when:

• The result will be used to make or defend a decision (ship the feature, keep the policy, publish the finding).
• You or the requester have a stake in the result coming out a particular way.
• The analysis is causal — identification claims are exactly where specification search does the most damage.
• The work will be scrutinized — a referee, a regulator, a skeptical exec, a replication.

If none of those hold, you're exploring; label it exploratory and move on.

What the plan locks (before seeing outcomes)

1. Hypotheses. Stated directionally and ranked: primary vs. secondary. Test the primary one; treat everything else as secondary or exploratory. Pre-commit the multiple-comparisons correction for secondary tests (Bonferroni / Benjamini–Hochberg / etc.) — uncorrected, they reintroduce the forking-paths problem the PAP exists to stop.
2. Estimand. The exact quantity (ATE/ATT/LATE/ITT), on the exact population, over the exact window. Reuse the question-framing brief.
3. Primary specification. One pre-committed model: functional form, controls, fixed effects, SE structure (and clustering level), inference method — the number you will report. Robustness specs support it; they don't replace it when you like them better. Write the primary specification to the presentation standard in causal-identification — equation with subscripts defined, level of variation, intuition, literature precedent — not an estimator name-drop.
4. Sample and exclusions. Inclusion criteria, exclusion rules, and how outliers and missing data are handled — decided now, by rule, not later by eye. "Drop obvious outliers" after seeing the data is a degree of freedom; "drop values beyond 3 IQR, pre-committed" is a rule.
5. Robustness suite. The alternative specs, placebo/falsification tests, and sensitivity analyses you commit to run regardless of whether the primary result survives them — you can't quietly drop the ones that disagree. Keep it small and targeted: two or three checks that probe the load-bearing assumption, not a catalogue; a pre-registered buffet is still a buffet. (causal-identification's design-specific diagnostics are mandatory and separate from this discretionary suite; "small" governs the discretionary specs only.)
6. Decision rule and power. What result leads to what action, and what counts as the effect being absent — define the null outcome so a null is a finding, not a prompt to keep digging. Commit too to the minimum detectable effect / power the design has: a null from an underpowered test isn't evidence of no effect, and saying so up front stops a noisy null from being read as a clean one.