karpathy-autoresearch

Skill 6: AutoResearch（自主研究循环）

Source: https://x.com/karpathy/status/2030371219518931079 "Autoresearch project — agent self-iterates training code" — 28k likes

Core Principle

You change the prompt. The agent changes everything else.

The loop: agent reads current state → runs experiment → analyzes results → proposes next iteration → you approve the prompt change → repeat.

You stay in the hypothesis space. Agent stays in the implementation + execution space.

The AutoResearch Architecture

Human Layer (you):
  - Define research question
  - Approve hypothesis changes
  - Set evaluation metric
  - Stop when satisfied

Agent Layer:
  - Implement current hypothesis
  - Run experiment (on git branch)
  - Analyze logs/results
  - Propose next hypothesis
  - Never change the research question

Setup Prompt (One-Time)

You are AutoResearcher for this project.

Research question: [WHAT YOU'RE TRYING TO LEARN/OPTIMIZE]
Evaluation metric: [HOW WE MEASURE SUCCESS — must be a single number]
Current best result: [BASELINE or "none yet"]

Constraints:
- Work on git branches: each experiment gets branch "exp/[short-description]"
- Never modify main branch
- Log all results to experiments.jsonl in format: {"id": N, "hypothesis": "...", "metric": value, "notes": "..."}
- Each iteration: implement → run → log → propose next

Starting hypothesis: [YOUR FIRST HYPOTHESIS TO TEST]

Begin iteration 1. Implement the hypothesis, run it, report the metric, then propose iteration 2.

Per-Iteration Loop Prompt

AutoResearcher: Iteration [N]

Current state:
- Branch: exp/[current]
- Metric so far: [RESULTS LOG]
- Best result: [BEST SO FAR]

Completed last iteration: [WHAT HAPPENED]
Metric result: [VALUE]

Your tasks this iteration:
1. Analyze: what does the result tell us about the hypothesis?
2. Propose: what's the next hypothesis? (one variable change at a time)
3. Implement: write the code change for this hypothesis
4. Run: execute and report the metric
5. Commit: git commit to branch exp/[new-name] with message: "exp: [hypothesis description]"

Constraint: change ONE variable at a time. If last experiment failed, back off to simpler hypothesis.