tune-eval

tune-eval — the honest scoreboard

Evaluation answers one question: does the tuned model meet the pre-registered bar on data it has never influenced? The validation set already steered training; only test.jsonl — untouched until now — gives an honest number (see concepts/validation-vs-test.md — bundled at the plugin root, ../../concepts/ relative to this file).

Teaching note: each step below is framed as four short lines before running it — What we're doing · Why (the failure it prevents) · Expect (healthy output) · Read (how to interpret what came out) — and one line after connecting result → next decision. One-liners, not essays; define jargon inline on first use with a concepts/ pointer. If the user says "skip the teaching" (or is clearly expert), drop Why/Expect/Read and keep What plus the result reading.

<skill-dir> below = the directory containing this SKILL.md; run commands from the user's project workdir.

Step 0 — Read the project state from disk (before asking anything)

On invocation, BEFORE asking the user a single question, check the project workdir:

1. EXPERIMENT-LOG.md — prior decisions, the tune-decide interview summary, training runs, and (critically) whether a bar + metric set was already pre-registered. tune-decide writes the interview and level decision there precisely so later skills — and later sessions — never re-ask. If a bar exists, confirm it in one line; do not renegotiate. Also check whether the current test.jsonl was already spent by a previous eval round.
2. runs/*/state.json — the run-continuity contract (tune-train owns writing it; all skills may read it):

{ "run_id", "status": "running|interrupted|completed|failed", "pid", "command",
  "model", "adapter_path", "data_dir", "log_path", "total_iters", "save_every",
  "hparams": {"batch_size", "learning_rate", "num_layers", "max_seq_length"},
  "started_at", "updated_at", "best_val": {"iter", "loss"}, "resume_history": [] }

This file hands you model, adapter_path, and data_dir — build every command below from it instead of asking. If status is running, do not eval a moving target: poll the log tail (tail runs/<id>/train.log), never hold the training process in conversation context. If interrupted, route back to tune-train first — resume is weights-only in mlx-lm 0.31.3 (--resume-adapter-file restores weights; fresh optimizer, iter counter resets): completed iters = highest NNNNNNN_adapters.safetensors in adapter_path; rerun with --iters <total minus completed> plus that checkpoint, and expect a brief loss bump from cold optimizer state. Only completed runs get the scoreboard.