training-run-hygiene

Training Run Hygiene

A completed run is not a healthy one. Every rule here exists to make "this model is not learning" visible while the run is still cheap to stop.

Persist A Scale-Free Training History

Every scientific training run must persist an append-only, machine-readable training history from the first optimizer update through every resume. Record at minimum:

• update/pass index, total loss, each unweighted loss component and its weighted contribution;
• learning rate, gradient norm, elapsed time;
• the exact source/unit coverage of the consumed prefix;
• training and validation curves separately when a validation objective exists.

Every loss component must also carry, in the same row, the mean square of its own target over its own element set, and the resulting scale-free ratio, where 1.0 is what predicting zero scores. A loss in physical units may never be reported as evidence that a model fits, because a small absolute error and a small target are indistinguishable without the denominator: a run whose loss read 0.0048 in physical units squared, against a target mean square of 0.0061, had learned almost nothing and looked healthy for a month.

Render human-readable curves at a predeclared cadence throughout training, not only at completion, and render them so a failure to learn is visible: plot the scale-free ratio with its no-skill reference line and a smoothed trend. One point per update with one source per point is dominated by source-to-source target scale and hides a trend of tens of percent.

Resume must append without gaps or duplicated steps. The launcher must fail before heavy training when the history schema, output path, or plotting cadence is absent.

If a frozen live run lacks a true loss history, add only a versioned checkpoint sidecar and label its metrics as checkpoint QC rather than optimizer-loss curves. Never reconstruct or imply losses that were never recorded.

Verify Mechanics, Then Train On The Full Eligible Dataset