autoresearch-ml

Autoresearch ML: Autonomous LLM Training Optimization

An autonomous experiment loop for single-GPU LLM pretraining. Edit train.py → commit → run 5-minute training → measure val_bpb → keep improvement or revert → repeat forever.

This skill is self-contained — it includes everything needed to set up and run the loop.

Setup Phase

1. Copy Template Assets

Copy the bundled training template to the project directory:

cp ${CLAUDE_SKILL_DIR}/assets/prepare.py .
cp ${CLAUDE_SKILL_DIR}/assets/train.py .
cp ${CLAUDE_SKILL_DIR}/assets/pyproject.toml .
cp ${CLAUDE_SKILL_DIR}/assets/program.md .

2. Install and Prepare

uv sync                    # Install dependencies
uv run prepare.py          # Download data shards, train tokenizer (~2 min)

3. Verify GPU

nvidia-smi
python -c "import torch; print(f'CUDA: {torch.cuda.is_available()}, Device: {torch.cuda.get_device_name()}, VRAM: {torch.cuda.get_device_properties(0).total_mem / 1e9:.1f} GB')"

4. Initialize the Experiment Session

1. Create a branch: git checkout -b autoresearch/<tag>-<date> — use $ARGUMENTS as the run tag if provided, otherwise propose one based on today's date
2. Gitignore the living session files — critical: git revert fails if autoresearch.jsonl is tracked, and if autoresearch.md (which you update mid-loop) is tracked, a revert can erase your learnings:

   printf '%s\n' autoresearch.jsonl autoresearch.md autoresearch.ideas.md run.log >> .gitignore
   git add .gitignore && git commit -m "autoresearch: add session files to gitignore"
   

3. Read prepare.py and train.py thoroughly to understand the codebase
4. Write autoresearch.md — a living session document recording goal, metrics, files in scope, constraints, and learnings. It is gitignored (living state), never committed or reverted.
5. Write autoresearch.sh — the benchmark script (see Benchmark Script section below)
6. Commit only autoresearch.sh (the immutable benchmark harness) — not autoresearch.md/.jsonl (gitignored above). train.py stays in history as normal.
7. Run baseline: bash autoresearch.sh
8. Parse metrics from output (lines matching METRIC name=value)
9. Record baseline in autoresearch.jsonl:
   • First write a config header: {"type":"config","name":"Optimize val_bpb","metricName":"val_bpb","metricUnit":"bpb","bestDirection":"lower"}
   • Then a status marker: {"type":"status","state":"running","timestamp":...}
   • Then record the baseline result
10. Begin the experiment loop