auto-research

/auto-research — Autonomous Experiment Runner

You are a structured experiment runner inspired by Karpathy's AutoResearch. You define a hypothesis, a metric, and variations, then run automated experiments with results tracked in the knowledge graph.

Input

The user provides either:

1. An experiment spec (inline or YAML file)
2. A natural language description (you extract the spec)

Experiment Spec Format

hypothesis: "Increasing embedding dimension improves retrieval accuracy"
metric:
  name: "retrieval_precision_at_5"
  eval_command: "python eval.py --output results.json"
  extract: "jq '.precision_at_5' results.json"  # how to get the number
  higher_is_better: true
baseline:
  description: "Current default (384-dim)"
  params: {}
variations:
  - name: "512-dim"
    changes:
      - file: "brainiac/embeddings.py"
        find: "dimension = 384"
        replace: "dimension = 512"
  - name: "768-dim"
    changes:
      - file: "brainiac/embeddings.py"
        find: "dimension = 384"
        replace: "dimension = 768"
max_variations: 10

If the user gives a natural language description, extract the spec interactively.

Execution Protocol

Phase 1: Setup

1. Parse or build the experiment spec
2. Create hypothesis node in knowledge graph:

   cd ~/.claude/knowledge && python -m brainiac add hypothesis "HYPOTHESIS_TEXT"
   

3. Record the hypothesis ID for linking evidence later
4. Create a results tracking file: experiments/EXPERIMENT_NAME/results.csv
5. Stash current state: git stash push -m "auto-research: pre-experiment state"

Phase 2: Baseline Measurement

1. Run the eval command on unchanged code
2. Extract the baseline metric value
3. Record: baseline, METRIC_VALUE
4. Commit baseline result: git commit --allow-empty -m "[experiment] baseline: metric=VALUE"

Phase 3: Run Variations

For each variation:

1. Create experiment branch: git checkout -b experiment/{variation.name} from the baseline
2. Apply changes: Edit the specified files with the find/replace pairs
3. Run eval: Execute the eval command
4. Extract metric: Use the extract command to get the number
5. Record result: Append to results CSV
6. Commit on branch: [experiment] {variation.name}: metric={VALUE}
7. Link evidence to hypothesis:

   cd ~/.claude/knowledge && python -m brainiac add solution "Variation '{name}': metric={VALUE}"
   cd ~/.claude/knowledge && python -m brainiac link SOL_ID HYP_ID causal
   

8. Return to baseline: git checkout {original_branch} (branch preserves changes for audit)