experimental-design

Experimental Design Skill

Design rigorous, reproducible experiments across scientific disciplines.

When to Use

• "Design an experiment to test..."
• "How many samples do I need?"
• "What controls should I include?"
• "Help me plan a clinical trial"
• "Is this experimental design valid?"
• Power analysis and sample size calculation

When NOT to Use

• Running the actual experiment (use code-execution)
• Analyzing collected data (use scipy-analysis + statsmodels-stats)
• Writing up results (use paper-writing)
• Literature review (use literature-search)

Design Components

1. Research Question and Hypotheses

• State clear, testable research question
• Formulate H0 and H1 (see hypothesis-gen skill)
• Define primary and secondary outcomes

2. Study Design Selection

Design	When to Use	Strengths	Weaknesses
RCT	Causal inference needed	Gold standard causality	Expensive, ethical limits
Factorial	Multiple factors	Tests interactions	Complex analysis
Crossover	Within-subject comparison	Reduced variability	Carryover effects
Quasi-experimental	Randomization impossible	Practical feasibility	Weaker causality
Observational (cohort)	Long-term outcomes	Natural setting	Confounding
Case-control	Rare outcomes	Efficient for rare events	Recall bias

3. Power Analysis

# Sample size calculation template (using scipy/statsmodels)
from statsmodels.stats.power import TTestIndPower
analysis = TTestIndPower()
n = analysis.solve_power(
    effect_size=0.5,   # Cohen's d (small=0.2, medium=0.5, large=0.8)
    alpha=0.05,         # Significance level
    power=0.80,         # Statistical power (commonly 0.80 or 0.90)
    ratio=1.0,          # Ratio of group sizes (n2/n1)
    alternative='two-sided'
)
print(f"Required sample size per group: {int(n) + 1}")

Key parameters:

• Effect size: Expected magnitude of difference
• Alpha: Type I error rate (usually 0.05)
• Power: 1 - Type II error rate (usually 0.80-0.95)
• Attrition: Add 10-20% for expected dropout