auto-research

Auto Research

Overview

Turn one testable idea into reproducible evidence, then produce a submission-ready artifact. When the task needs new autoresearch training evidence, run the official VESSL cookbook recipe on one approved A100 with a bounded safety and W&B observability overlay. Keep claims no stronger than the recorded results.

Read the event workflow and evidence contract before planning. For a Training campaign, also read the complete official VESSL autoresearch runbook. Copy the appropriate template from assets/ into the participant workspace.

Preflight

Confirm the selected Track, current time, 16:30 submission hard cut, permitted public data, available evidence, and whether the work creates new autoresearch training evidence. Do not load private participant, reviewer, messaging, or operations records.

Submission Contract

Path	Required output
Track 1	Agent workflow plus a 2-4 page workshop-style short paper and self-review
Track 2	Review agent plus an ICML-style review result for a Track 1 paper
Both	Complete Track 1 evidence first, then review the frozen paper with Track 2

Treat the final paper/agent submission at 16:30 as a hard cut. Peer and self-review may follow. During the Ralph Loop, operate through the coding agent and preserve an audit trail of prompts, code, runs, and outputs.

Workflow

Select exactly one path below. Do not merge the compute and no-compute preflight requirements. Use the bundled references, assets, and recorder directly from this skill.

Select exactly one path

Training path (Track 1 or Both)

Choose this path only when generating new autoresearch training evidence. It includes all compute, cost, metric, and onboarding requirements below.

1. Freeze a research spec containing one falsifiable hypothesis, a named baseline, val_bpb as the evaluation metric, dataset, budget, and stop condition.
2. REQUIRED SUB-SKILL: Use wandb-onboarding, including its synthetic offline run. Before online sync, show entity, project, visibility, and the exact W&B allowlist and obtain explicit confirmation.
3. REQUIRED SUB-SKILL: Use vessl-cloud-onboarding. Run vesslctl resource-spec list --usable-only -o json; show the live exact single A100 spec, hourly price, credit, image, object volume, wall-clock cap, total estimate, and cleanup/timeout plan. Provision only after explicit confirmation.
4. Use vessl-ai/vessl-cloud-cookbook/autoresearch pinned at 97a0af14b0acae042162b1f70f17fbe2d570afa2 as the execution SOT. Use a participant-owned writable fork and the recipe's vesslctl job create flow. Override its default with the approved live A100 spec; do not fall back to another GPU, CPU, or a larger model, and never substitute H100.
5. Keep the official benchmark unchanged: this is an unchanged benchmark, not a reimplementation. The baseline uses the pinned prepare.py and train.py; candidates may modify only train.py, one hypothesis and one change at a time. Do not modify the evaluation harness, dependencies, benchmark reports, data, tokenizer, or batch-job scripts.
6. Copy the campaign control file, VESSL A100 run card, experiment ledger template, and scripts/record_experiment.py, then execute the complete official VESSL autoresearch runbook. The recorder consumes each fetched VESSL log locally, appends experiments.jsonl, and creates only an allowlisted W&B offline run until separately approved sync.
7. Run one baseline, at most three candidate trials executed sequentially, and one winner confirmation. Do not use the cookbook's parallel fan-out or unbounded loop. Keep only a lower val_bpb.