watch-runs

/watch-runs — caretaker loop for active runs

Watch every non-terminal run of this project until all reach a terminal state, without the user having to ask "怎么样了".

Protocol

1. Inventory. mcp__projtool__list_run_statuses(experiment_id=...) for each experiment with active runs (start from mcp__projtool__list_runs if unsure). Note each run's phase, progress_age_s, and stalled_suspect.
2. Close what already exited. Newly exited runs (exit sentinel present): mcp__projtool__finalize_runs(experiment_id=..., then=<policy>) — one call per experiment, one journal commit per batch.
3. Report briefly. One line per active run: run_id tail, phase, progress age. Flag stalled_suspect=true runs — check their progress_sources before declaring a stall (pool launchers with declared progress_paths often show a fresh source behind a quiet parent log).
4. Schedule the next check with ScheduleWakeup (or your harness's scheduler): first check ~5 min after launch, then 15 → 30 → 45 min as runs prove stable. NEVER busy-wait or run while true; do ssh loops.
5. Repeat 1–4 on each wake until no non-terminal runs remain.
6. Finish. Summarize outcomes (per-status counts, headline metrics from compare_runs), confirm the instance policy took effect (list_instances — nothing billing idle), and suggest /retrospective to distill findings.

Completion by artifact, not just process exit

Some runs signal "done" by dropping a file — metrics.json, a .ok/.done sentinel — rather than by the process exiting, so their exit sentinel may never land on the schedule you expect. For those runs, treat the expected artifact's presence as the completion signal.

On EACH scheduled wake (step 4), probe for it once — a single mcp__projtool__read_remote_file(...) or mcp__projtool__remote_exec(command="test -f <path> && echo DROPPED") call, using the run's declared outputs / progress_paths when present. One probe per wake, then let the schedule pace you.