autoresearch

Autoresearch: Autonomous Experiment Loop

An autonomous optimization loop where Claude edits code, runs a benchmark, measures a metric, and keeps improvements or reverts — repeating forever until stopped.

Core Concept

The loop is simple: edit → commit → run → measure → keep or discard → repeat.

• Primary metric is king. Lower (or higher, depending on direction) is better. Improved → keep the commit. Equal or worse → git revert.
• State survives context resets via autoresearch.jsonl (append-only log) and autoresearch.md (living session document).
• Domain-agnostic. Works for any measurable target: test speed, bundle size, LLM training loss, Lighthouse scores, build times, etc.
• Be careful not to overfit to the benchmarks and do not cheat on the benchmarks. Optimize the real workload, not the measurement harness.

Setup Phase

When the user triggers autoresearch, gather the following (ask if not provided). If $ARGUMENTS is provided, use it as the optimization goal:

1. Goal — what to optimize (e.g., "reduce unit test runtime") — use $ARGUMENTS if provided
2. Command — the benchmark to run (e.g., pnpm test, uv run train.py)
3. Primary metric — name, unit, and direction (lower or higher is better)
4. Secondary metrics — optional additional metrics to track for tradeoff monitoring (e.g., memory, compile time)
5. Files in scope — which files can be modified
6. Constraints — time budget, off-limits files, correctness requirements

Optionally check for .claude/autoresearch-ai-plugin.local.md in the project root for persistent configuration:

---
enabled: true
max_iterations: 50
working_dir: "/path/to/project"
benchmark_timeout: 600
checks_timeout: 300
---

# Autoresearch Configuration

Additional context or notes for this project's autoresearch setup.

• enabled — whether autoresearch is active (default: true)
• max_iterations — stop after N experiments (default: 0 = unlimited)
• working_dir — override directory for experiment files (default: current directory)
• benchmark_timeout — benchmark timeout in seconds (default: 600)
• checks_timeout — correctness checks timeout in seconds (default: 300)