experiment-pipeline

Experiment Pipeline

A structured 4-stage framework for executing research experiments from initial implementation through ablation study, with attempt budgets and gate conditions that prevent wasted effort. This follows the Experiment Tree Search design from the EvoQuant paper, where the engineer agent iteratively generates executable code, runs experiments, and records structured execution results at each stage.

When to Use This Skill

• User has a planned experiment and needs to organize the execution workflow
• User wants to systematically validate a novel method against baselines
• User asks about experiment stages, attempt budgets, or when to move on
• User needs to reproduce baseline results before testing their method
• User mentions "experiment pipeline", "baseline first", "ablation study", "stage budget", "experiment execution"

The Pipeline Mindset

Experiments fail for two reasons: wrong order and no stopping criteria. Most researchers jump straight to testing their novel method without verifying their baseline setup, then wonder why results don't make sense. Others spend weeks tuning hyperparameters without a budget, hoping the next run will work.

The 4-stage pipeline solves both problems. It enforces a strict order (each stage validates assumptions the next stage depends on) and assigns attempt budgets (forcing systematic thinking over brute-force iteration).

Before Starting: Load Prior Knowledge

If coming from research-ideation, your research proposal (Step 7) provides the experiment plan — datasets, baselines, metrics, and ablation design — that maps directly to Stages 1-4 below.

Dataset rule (applies to every stage): the proposal's dataset plan is already scoped to what is actually available, not the paper's dataset. The local dataset usually differs from the paper's (universe, date window, vendor). So reproduction is a faithful reconstruction on the scoped local data, not an exact match to the paper's numbers. Compare against a tolerance justified by the documented dataset differences; reserve "within 2% of reported" for the rare case you run on the paper's own data.