experiment-plan

Experiment Plan

Design a complete experiment plan that a reviewer would call "thorough." No hand-waving, no "we'll figure it out later."

Decision Gate

Before running:

• Research question is crystallized (refine-logs/RESEARCH_QUESTION.md exists)
• Baselines are identified (≥8 per quality standards)
• Compute resources are known (GPU type, hours available)
• Datasets are accessible

Phase 1 — Experiment Block Design

Design 5-7 experiment blocks, each answering one sub-question:

1. B1: Phenomenon validation — Does the claimed phenomenon exist? (Sanity check)
2. B2: Ablation / isolation — Is our method responsible, not confounders?
3. B3: Method comparison — Head-to-head vs all baselines on primary metrics
4. B4: Mechanism analysis — Why does it work? (Interpretability, probing)
5. B5: Robustness — Does it hold across domains/scales/perturbations?
6. B6: Downstream impact — Does improvement on proxy metric translate to real value?
7. B7: Extended / realistic — Real-world simulation or deployment scenario

For each block:

• Hypothesis (falsifiable)
• Metrics (primary + secondary)
• Expected outcome range
• Failure mode (what would disprove the hypothesis)

Output: refine-logs/EXPERIMENT_PLAN.md (blocks section)

Phase 2 — Baseline Specification

For each of the 8+ baselines:

Baseline	Paper	Year	Implementation	Status
...	...	...	official/reimpl/ours	available/needed

• Verify implementation availability (GitHub links, paper repos)
• Note any baselines that need reimplementation (flag as risk)
• Ensure fair comparison: same data splits, same preprocessing, same compute budget

Phase 3 — Milestone & Decision Gate Design

Define sequential milestones with kill conditions:

M0 (sanity)     → M1 (phenomenon?) → M2 (full panel) → M3 (comparison) → M4 (mechanism) → M5 (robustness) → M6 (extended)

Decision gates:

• M1 gate: If phenomenon effect size < threshold → STOP or pivot
• M3 gate: If our method not statistically significant vs best baseline → fall back to analysis paper
• M5 gate: If robustness fails on >50% of perturbations → scope down claims