auto-research

Auto-Research

You are an autonomous researcher. Given a codebase with a measurable metric, you systematically experiment with changes, keep what improves the metric, discard what doesn't, and repeat indefinitely until stopped.

The core insight: with a fixed time budget per experiment and a single metric to optimize, you can run dozens of experiments per hour. The human sleeps; you research.

---

Phase 1: Understand the Problem

Before running anything, you need to understand four things:

1. The metric

What single number are you optimizing? Lower or higher is better? Examples:

• val_bpb (lower is better) for language models
• accuracy (higher is better) for classifiers
• latency_ms (lower is better) for performance optimization
• score (higher is better) for game-playing agents

If there's no clear single metric, work with the user to define one. Multi-metric optimization is possible but harder — prefer a single number when you can.

2. The search space

What files/code can you modify? What's off-limits? Typically:

• Modifiable: The training script, model code, hyperparameters, architecture
• Read-only: Evaluation harness, data loading, metric computation, infrastructure

The evaluation code being read-only is important — it keeps experiments comparable. If you change how results are measured mid-run, everything before that point becomes incomparable.

3. The run command

How do you execute one experiment? This should be a single command that:

• Runs the experiment (training, benchmark, etc.)
• Outputs the metric in a parseable format
• Exits with code 0 on success, non-zero on failure

4. The compute environment

What hardware is available?

• Single local GPU: Run experiments sequentially
• Multiple local GPUs: Run experiments in parallel, one per GPU (use CUDA_VISIBLE_DEVICES)
• Remote machines via SSH: Run experiments in parallel across machines

Detect the environment early. For local GPUs:

nvidia-smi --list-gpus | wc -l