autoresearch-setup

Autoresearch Setup

Generate a complete autonomous research loop for any domain. Based on the autoresearch philosophy: an AI agent modifies code, runs experiments on a fixed budget, keeps or discards based on a single metric, and repeats indefinitely.

When to use: User wants to set up autonomous experimentation for ML training, kernel optimization, prompt engineering, hyperparameter search, or any domain where iterative improvement can be measured by a single metric.

Core Philosophy

1. Single mutable file — the agent only edits ONE file. Everything else is fixed.
2. Fixed time budget — every experiment runs for the same wall-clock time, making results directly comparable.
3. Single metric — one number decides keep/discard. Lower or higher, pick one direction.
4. program.md as research org code — the human writes strategy in Markdown, not Python. The agent interprets and executes.
5. Git-backed experiments — every change is a commit. Keep = advance branch. Discard = reset.
6. Results in TSV — plain text, human-readable, machine-parseable.
7. Never stop — the agent runs autonomously until manually interrupted.
8. Simplicity criterion — if equal performance, simpler code wins. Ugly complexity for tiny gains is not worth it.

Setup Flow

When the user asks to set up autoresearch for a given context, follow these steps:

Step 1: Understand the Domain

Ask or infer:

• What is being optimized? (model architecture, kernel, prompt, config, algorithm...)
• What is the metric? (loss, accuracy, throughput, latency, score...)
• Metric direction? (lower is better / higher is better)
• Time budget per experiment? (default: 5 minutes)
• What hardware/environment? (GPU, CPU, cloud, local...)
• What are the fixed constraints? (data, evaluation, dependencies)

Step 2: Generate Project Structure

Create this structure in the target directory:

<project>/
├── program.md        — Agent instructions (human-edited strategy)
├── prepare.py        — Fixed: data prep, evaluation, constants (DO NOT MODIFY)
├── experiment.py     — Mutable: the file the agent edits (ONLY THIS FILE)
├── results.tsv       — Experiment log (git-ignored)
├── pyproject.toml    — Dependencies (locked)
└── .gitignore        — Ignores results.tsv, run.log, __pycache__