experiment-design

Experiment Design

Overview

Good experimental design determines whether your results are interpretable, valid, and reproducible. This skill covers the process from hypothesis formation through post-experiment reporting, with actionable checklists and code examples.

When to Use This Skill

• Before designing any experiment: choose baselines, define metrics, plan sample sizes
• When writing a research idea: include a concrete experimental protocol
• When results look unexpected: check whether experimental design is the cause
• When a reviewer asks about confounds, baselines, or statistical power

---

Hard Rules

1. Define your hypothesis BEFORE collecting data — post-hoc hypothesis selection inflates false positive rates
2. Always include at least one trivial baseline — random, majority class, or simple heuristic
3. Set random seeds everywhere — numpy, random, torch, tensorflow, sklearn
4. Use held-out test data only once — never tune on it
5. Report N (sample size) and how it was determined — underpowered studies mislead

---

1. Hypothesis Formation

A well-formed hypothesis has three parts:

Part	Example
Null hypothesis (H₀)	The proposed method performs no better than the baseline
Alternative hypothesis (H₁)	The proposed method achieves higher F1 than the baseline
Expected effect size	Δ = 0.03 F1 (3 percentage points)

Directionality:

• One-tailed: you predict direction of effect (method > baseline) → more statistical power, but only justified if direction is theoretically grounded
• Two-tailed: you test for any difference → safer for exploratory work

---

2. Statistical Power and Sample Size

Power analysis determines how many samples you need to detect an effect of a given size.

Key parameters:

• α (Type I error rate): probability of false positive — typically 0.05
• β (Type II error rate): probability of false negative — typically 0.20
• Power = 1 - β = 0.80 (minimum), 0.90 (preferred)
• Effect size: domain-specific (Cohen's d for continuous, Cohen's h for proportions)