run-experiment

run-experiment

Submit a training job to the GPUPlane scheduler and confirm it starts. This is step 1 of the experiment loop (run → monitor → analyze). It returns immediately with a task handle — training runs for minutes to hours; do not wait inline.

Workflow

1. Check the fleet — list_nodes to see which GPUs are online and free. Pick a node by label if the user specified one (e.g. rtx5090). If every GPU is busy, say so and stop — do not queue behind an unknown wait without telling the user.
2. Submit — submit_job with the user's command, the project's absolute working_dir (on the agent host), gpus, and a name. If the user gave a project/experiment name, pass it. Keep priority=100 unless told otherwise.
3. Confirm dispatch — poll get_job(job_id) every ~15s (use the pollIntervalMs from the handle) until status moves from QUEUED → RUNNING. If it stays QUEUED for minutes with free GPUs, something is wrong with the agent — say so.
4. Hand off to monitoring — once RUNNING, a run exists; get its id from get_job(job_id).run_id, then call get_run_summary(run_id) and tell the user "training started, run_id=… — I'll watch it." Then invoke the monitor-experiment skill (or let the user call it).

Conventions (repo AGENTS.md)

• The training script must log metrics via the GPUPlane SDK or TensorBoard so the control plane can see train/loss, eval/loss, … Without metrics, monitoring/anomaly tools have nothing to read.
• Register a primary metric with set_primary_metric before relying on recommendations — get_best_checkpoint ranks by the experiment selection, then project selection, then global default. Convention: eval/loss (minimize) or eval/accuracy (maximize).
• Never submit_job with command=["python","-i"] or anything interactive — the agent runs headless; an interactive process hangs forever and wastes the slot. The command must exit on its own.