ml-research-workflow

ML research workflow

projtool runs ML experiments across a local development machine and remote AutoDL containers. Three asset categories — code, outputs, reports — have different version-control needs and different storage backends. Your job is to keep them coherent without making the user do bookkeeping.

Hard policy: remote-first, watched, in the user's language

Remote-first (user ruling 2026-07-16). Experiment workloads — training, evaluation, data preprocessing, dataset downloads — run on the remote instance, never on the user's machine. The local machine is for code editing, report writing, and light analysis (seconds of CPU, no GPU, no bulk disk I/O). If the user explicitly asks for a local run, warn about CPU/disk impact and run AT MOST ONE at a time. If remote throughput is the bottleneck, scale remotely — start_runs(parallelism=N) lanes or another slot — instead of spilling to the local box. Do not self-ration to save money (user ruling: "我不缺钱"): renting another instance or a bigger card is always preferable to queueing work locally or serializing what could run in parallel. Cost forks (wait for stock vs take a pricier spec) are the user's call — present the options; don't silently pick the cheap one.

Long remote operations (>~2 minutes). Never run them as a blocking MCP call. Anything that installs, downloads, trains, or converts for minutes goes through run_script (tracked, cancellable, watchable); remote_exec is for commands that finish in seconds. Two non-obvious rules learned from incidents:

• A cancelled long call did NOT stop. Cancelling a blocking MCP call only abandons the client side — the remote command keeps running and may complete minutes later (2026-07-16: a cancelled 40-minute env setup finished on its own and was recorded as a success). After any cancel of a long projtool call, assume the remote side is still executing: check state (get_run_status, a short remote_exec probe, diagnose) before retrying, or you will run the operation twice.
• Re-launching after an interruption must be idempotent. After an API stall, session restart, or a user "继续", call list_run_statuses FIRST and reconcile against what is already running/exited. Blind re-launch duplicates runs and leaves orphans nobody finalizes (2026-07-17: two orphan runs from exactly this).