autoresearch

Autoresearch

Philosophy

Bounded evidence loop: baseline, hypothesize, patch, score, decide, record. Humans set goal, metric, scope, and stop condition; the agent runs reversible hypotheses inside those bounds.

When To Use

• The user explicitly names $autoresearch.
• The user asks to set up, run, refine, or audit an Autoresearch-style loop.
• The work has target path, editable boundary, metric, verifier, and stop condition.
• The user asks whether a tiny delta, protected regression, or evaluator/data/cache edit is safe to keep.

Avoid

• Generic feature work, keeping unverified experiment changes, or editing fixed harness/evaluation surfaces unless the user changes the benchmark contract.

Execution Boundaries

Owns the experiment contract, ledger, and keep/discard/block recommendation; parent thread owns final decision. Fixed surfaces are benchmark harness, evaluator, data prep, datasets, tokenizer files, and guard commands. Block on unclear metric, boundary, runtime, guard semantics, network/dependency/destructive approvals, contract edits, or unbounded runs.

Inputs

Target path, boundaries, run tag, metric direction, verify/guard commands, stop condition, evidence path, train/selection/test split policy, and optional evaluator contract or min_delta policy.

Deliverables

Ledger plus closeout: hypotheses, patches, commands, scores, baseline, best delta, guard status, changed files, blockers, and schema_version when schema-bound. For skill optimization contracts, also produce best_skill.md, rejected-edits.jsonl, and promotion.json before recommending a canonical edit.

Discovery Interview

• Ask one round at a time.
• Use a plain-language question.
• Explain why this matters for the current skill decision.
• Avoid dumping the whole interview plan at once.
• Read references/discovery-interview.md when the request is underspecified.

Workflow

1. Confirm target, instructions, boundaries, run tag, stop condition, and evidence paths.
2. Require goal, scope, metric direction, verify command, and stop condition.
3. For jscraik/autoresearch, read README.md, program.md, prepare.py, and train.py; normally edit only train.py.
4. Define parser contract, guard command, held-out checks, noise_runs, aggregation, min_delta, and confirmation rule.
5. Baseline first. Never keep an experiment before baseline evidence exists.
6. Before each iteration, re-read ledger, logs, git status, commits, and last kept diff.
7. Run one reversible hypothesis, Verify, optional Guard, then keep/discard/crash/block with evidence and update the ledger.
8. If attempts plateau, pivot using ledger and git history; at closeout, compare against the original rubric or metric.
9. When references/contract.yaml declares optimization.enabled, treat that block as the authority for split visibility, edit budget, protected paths, anti-cheat checks, and promotion. Write candidates under the evidence root; do not overwrite canonical SKILL.md until the promotion contract passes review.