training-monitor

Training monitor

A pretraining run fails in a small number of ways, and most of them show up in a handful of every-step numbers before the loss curve moves. This skill restates the metric set, the reduction rules, and the triage order from Deng (2026) so that an agent can read a run the same way each time. The source is explicit that "every lab has its own dashboard, so this list is necessarily opinionated"; treat it as a starting point that the run's own healthy history then calibrates.

Scope. Language-model pretraining first; the loss, gradient, system, tensor, and evaluation sections also apply to large SFT and RL runs with the obvious substitutions. The expert-balance section applies only to mixture-of-experts models, the modality section only to multimodal models. This skill contains no code: which checks exist, where they run, and whether they become a script is the reading agent's decision, made from the procedure below.

How to apply this to a project

1. Inventory before you instrument. List what the training stack already logs, namely the tracker keys, the stdout lines, and the evaluation outputs. Most stacks already emit the global loss, a global gradient norm, step time, tokens per second, and a held-out loss.
2. Map the inventory onto the table in the next section and mark each row as present, derivable, or missing. Derivable rows come from logged quantities without new hooks: co-spike count from the loss and gradient-norm histories, update ratio from update and weight norms when both are logged, tokens per second from token counts and step time, straggler score from per-rank step times.
3. Fill P0 gaps first and leave P1 and P2 for when a P0 signal misbehaves. A monitor that is complete but late is worth less than one that is partial and on time.
4. Set thresholds from healthy runs, never from one failure. Deng: "Choose thresholds from healthy runs. Do not tune the window length or the $z_t$ cutoff around one failure." Record the chosen thresholds next to the run config so the next run inherits them.
5. Pick the sampling interval $K$ from the cost rules in Part V, then measure the overhead on the actual model and hardware before adding anything.
6. Decide the tooling last. A check that repeats every run or chains into a gate deserves a script with an exit status. A single investigation deserves a query or a notebook. A short run deserves a person reading the dashboard. The doctrine is the same in all three.
7. When something looks wrong, walk the triage order in Part I and stop at the first step that explains the symptom; open per-layer charts only when the walk ends without an explanation.
8. Compare runs by tokens or FLOPs, not by step, whenever batch size or sequence length differs between them.