monitor-experiment

Monitor Experiment Results

Monitor: $ARGUMENTS

Workflow

Step 1: Check What's Running

Queue-batch context (when invoked as part of an /experiment-queue batch). If the caller invokes this skill with a job-ID that belongs to an active /experiment-queue batch, the batch's queue_state.json (under $REMOTE_RUN_DIR/queue_state.json on the SSH host) is the authoritative source for that job's status, started, and completed timestamps — the queue scheduler has already verified expected_output exists. Phase 3.6 below still runs (it's how cost.json gets finalized), but Step 1 / 2 / 3 may use queue_state.json to skip the screen -ls + log-parsing path for jobs the scheduler has already marked terminal. For standalone use (no active queue), Step 1's screen -ls is the canonical check.

SSH server:

ssh <server> "screen -ls"

Vast.ai instance (read ssh_host, ssh_port from vast-instances.json):

ssh -p <PORT> root@<HOST> "screen -ls"

Also check vast.ai instance status:

vastai show instances

Modal (when gpu: modal in CLAUDE.md):

modal app list         # List running/recent apps
modal app logs <app>   # Stream logs from a running app

Modal apps auto-terminate when done — if it's not in the list, it already finished. Check results via modal volume ls <volume> or local output.

Step 2: Collect Output from Each Screen

For each screen session, capture the last N lines:

ssh <server> "screen -S <name> -X hardcopy /tmp/screen_<name>.txt && tail -50 /tmp/screen_<name>.txt"

If hardcopy fails, check for log files or tee output.

Step 3: Check for JSON Result Files

ssh <server> "ls -lt <results_dir>/*.json 2>/dev/null | head -20"

If JSON results exist, fetch and parse them:

ssh <server> "cat <results_dir>/<latest>.json"

Step 3.5: Pull W&B Metrics (when wandb: true in CLAUDE.md)

Skip this step entirely if wandb is not set or is false in CLAUDE.md.