experiment-craft

Experiment Craft

A systematic approach to running, debugging, and iterating on research experiments. The critical skill is not running more experiments — it's understanding WHY experiments fail.

When to Use This Skill

• User's experiment is not working or producing unexpected results
• User needs help diagnosing why a method fails on certain data
• User wants to organize their experiment process with structured logging
• User asks about debugging research code or iterating on approaches
• User mentions "experiment debugging", "why doesn't this work", "experiment log", "results are wrong"

This skill is typically loaded from within experiment-pipeline when a stage attempt fails. After debugging, return to the pipeline's stage-gate structure to continue. Can also be used standalone for any experiment debugging.

The Debugging Mindset

Finding WHY experiments fail is the most critical research skill. Not analyzing results leads to two failure modes:

1. Slow progress: Running random experiments without understanding failure causes
2. Wasted time: Abandoning good approaches because activation tricks were missed

The goal is not to run more experiments. The goal is to run the RIGHT experiments — ones that isolate causes and test specific hypotheses.

5-Step Diagnostic Flow

When an experiment fails or produces unexpected results, follow these five steps:

Step 1: Collect Failure Cases

Gather concrete examples of bad results. Look at the actual outputs, not just aggregate metrics. What specifically went wrong? Are the failures systematic or random?

Step 2: Find a Working Version

You need a baseline that works. Two ways to find one:

• Simplify the task: Reduce data complexity, relax the task setting, add more supervision, use easier inputs
• Remove your changes: Start from the baseline method and remove your algorithmic improvements one by one

If you can't find any working version, simplify further until something works. There is always a simple enough version that works.