auto-experiment

Workflow 1.5: Experiment

Implement and deploy experiments from plan: $ARGUMENTS

Overview

This skill bridges Workflow 1 (idea discovery + method refinement) and Workflow 2 (auto review loop). It takes the experiment plan and turns it into running experiments with initial results.

Workflow 1 output:                    This skill:                                    Workflow 2 input:
refine-logs/EXPERIMENT_PLAN.md   →   implement → LLM review → deploy → collect → initial results ready
refine-logs/EXPERIMENT_TRACKER.md     code        (cross-model)    /run-experiment     for /auto-iteration-loop
refine-logs/FINAL_PROPOSAL.md

Constants

• RESEARCH_DOMAIN = auto — Project domain tag (free-form, e.g. mechanistic-interpretability, vision-encoders, rl-policy-eval). Consumed by Phase 1.5 only as a routing constraint to /mechanism-skills — see Phase 1.5 Step 2's domain: arg. When null or auto, Phase 1.5 infers from FINAL_PROPOSAL.md; on ambiguous inference, silently default to general and log [research-domain] inference ambiguous — defaulted to general (this fallback bypasses AUTO_PROCEED by design — see /auto's flag-table row for the canonical statement). To force a specific domain, pass it explicitly on the CLI. (Note: Phase 1.1 routes through /experiment-tips using its own symptom-level trigger table and does not consume this constant.)
• MECHANISM_ROUTING = auto — Phase 1.5 mechanism-family routing mode. auto (default): invoke /mechanism-skills, write refine-logs/MECHANISM_ROUTING.md, present 2–3 candidates and let the caller pick (auto-select #1 when AUTO_PROCEED=true; otherwise block on the caller's AskUserQuestion). skip: assume routing already exists (or is not applicable) and proceed. not-applicable: explicitly mark behavioral-only proposal and skip without invoking. When called from /auto, the orchestrator's mini-prompt fills CHOSEN_FAMILY so this skill is re-entered with MECHANISM_ROUTING=skip.
• CHOSEN_FAMILY = none (dynamic — not in config; forwarded by /auto's orchestrator — from MECHANISM=given (the user's chosen_mechanism captured by the claim stage), the AUTO_PROCEED=false family mini-prompt, or an explicit family: pin in task.md (cross-round Rule 2), after any settled-pin conflict is resolved) — When set, commits this family/submethod combo from MECHANISM_ROUTING.md before implementation (Phase 1.5 Mode B).
• CODE_REVIEW = true — external LLM reviewer checks experiment code before deployment. Catches logic bugs before wasting GPU hours. Set false to skip.
• AUTO_DEPLOY = true — Automatically deploy experiments after implementation + review. Set false to manually inspect code before deploying. Treated as a standing approval for the deploy step: when AUTO_DEPLOY=true, the Phase 4 deploy proceeds even if AUTO_PROCEED=false.
• AUTO_PROCEED = true — Whether the Phase 4 Experiment Gate may skip the UI prompt. When true (default) and AUTO_DEPLOY=true, the gate proceeds silently. When false and AUTO_DEPLOY=false, the gate calls AskUserQuestion (approve / narrow-scope / abort) and blocks until the user answers. AUTO_DEPLOY=true overrides AUTO_PROCEED=false for this gate (standing approval). Forwarded from /auto.
• SANITY_FIRST = true — Run the sanity-stage experiment first (smallest, fastest) before launching the rest. Catches setup bugs early.
• MAX_PARALLEL_RUNS = 4 — Maximum number of experiments to deploy in parallel (limited by available GPUs). For Phase 4's queue dispatch path (Phase 4.B), this becomes max_parallel: in the /experiment-queue manifest. For the direct dispatch path (Phase 4.A), it's the in-skill concurrency cap on /run-experiment calls.
• BATCH_DISPATCH = auto — Phase 4 dispatch routing rule. auto (default): per the Phase 4.0 table — milestones with ≥ 10 runs, depends_on, grid expansions, or ≥ 3-seed × ≥ 3-config multi-seed sweeps go to /experiment-queue; smaller ad-hoc milestones go to /run-experiment. queue: force every milestone to /experiment-queue (use when you know the workload benefits from OOM retry + stale cleanup even at small sizes). direct: force every milestone to /run-experiment (use only when debugging the queue scheduler itself; emits a warning if any milestone would have triggered the queue rule under auto). Forwarded from /auto.
• BASE_REPO = null — GitHub repo URL to use as base codebase. When set, clone the repo first and implement experiments on top of it. When null, write code from scratch or reuse existing project files.
• COMPACT = false — When true, (1) read idea-stage/IDEA_CANDIDATES.md instead of full idea-stage/IDEA_REPORT.md if available, (2) append experiment results to EXPERIMENT_LOG.md after collection.
• RESUME = false — When true, each phase checks if its primary artifact already exists non-empty and skips itself if so (see "Resume protocol" below). Useful for picking up after a crash. Default false = every phase always runs from scratch and overwrites prior artifacts. Resume never deletes pre-existing files.
• GPU_ID = auto — GPU device(s) to use for sanity and full-suite runs. auto (default) inherits from environment / launcher. A single id (0) or comma-list (4,5,6,7) causes Phase 3 (sanity) and Phase 4 (deploy) to pass CUDA_VISIBLE_DEVICES=<GPU_ID> as the first positional argument to /run-experiment — the run-experiment skill then exports this env var before launching the experiment subprocess (do not treat it as a shell prefix; /run-experiment is a Skill invocation, not a shell command). Also record the effective CUDA_VISIBLE_DEVICES into each run's run.sh so reproductions land on the same devices. Override: — gpu-id: 4,5,6,7. When GPU_ID lists multiple devices and MAX_PARALLEL_RUNS > 1, partition devices across concurrent runs (e.g., GPU_ID=4,5,6,7 + 2 parallel → run A on 4,5, run B on 6,7); do not co-schedule two runs on the same device unless memory measurements confirm fit. Forwarded from /auto and from agents/experiment.md; /auto-verify follows the same convention for verify variants.