experiment-design

Experiment Design Methodology

You are helping a researcher design rigorous experiments. Follow this methodology systematically.

Step 1: Understand the Research Question

Before designing any experiment:

• Ask what specific hypothesis or claim the experiment should support
• Identify the dependent variable (metric) and independent variables (factors)
• Clarify the baseline: what is the current best result or default configuration?

Step 2: Single-Variable Isolation

Every ablation study must change exactly ONE variable at a time. For each factor:

1. Define the factor — what is being varied (e.g., loss function, learning rate, architecture component)
2. List levels — all values this factor will take (e.g., CE, focal, VAR)
3. Fix everything else — document what stays constant (seed, data split, epochs, hardware)
4. Predict outcome — before running, state what you expect and why

Template for each ablation row:

| Run ID | Factor | Value | Fixed Config | Expected Outcome |
|--------|--------|-------|-------------|-----------------|

Step 3: Experiment Matrix

For multi-factor studies, use a structured matrix:

1. Full factorial — if factors are few (≤3) and levels are few (≤3 each)
2. Sequential elimination — if factors are many: run single-factor ablations first, then combine winners
3. Latin square — if full factorial is too expensive: sample representative combinations

Always calculate total runs before committing:

Total runs = product of all factor levels
GPU hours = total runs × hours_per_run

Step 4: Resource Estimation

For each experiment plan, estimate:

• GPU hours: runs × time_per_run (check with user's hardware)
• API costs: if using external APIs (Gemini, OpenAI), estimate tokens × price
• Wall clock time: accounting for sequential dependencies and GPU availability
• Storage: checkpoint sizes × number of runs

Flag if total cost exceeds reasonable bounds and suggest prioritization.