gate

gate — accept only real improvements, on val

The gate is where dishonest optimization is prevented. Search is a noise amplifier: try enough candidates and some will look better by chance alone (the more candidates you screen, the larger the expected best-of-noise). The gate is the rule that keeps a lucky draw from being promoted to "the new best". It refuses any split but val, and by default accepts a candidate only when its val reward beats the current best by more than k standard errors.

Inputs / outputs (manifest tokens)

• needs: scores — the candidate's and current best's val reward and stderr (from evaluate). The SE is not optional: significance is meaningless without it.
• provides: decision — {accept, reason, delta, threshold}, the audit record of why a candidate was kept or rejected.

The significance rule

paired (the default):   accept ⟺ mean(Δ[t]) > k · SE(Δ)     over the SAME val tasks
significant (fallback): accept ⟺ Δ = cand − curr > k · sqrt(cand_se² + curr_se²)

The bar is Δ > k·SE and not Δ > 0 because search is a noise amplifier: screen enough candidates and the best-looking one is best by luck, so Δ > 0 banks noise as progress and the val curve climbs while nothing improved. Clearing k standard errors of the measurement's own error is what makes an accept mean something — turn this down and the run's numbers stop being evidence. k=1 is lenient (~1σ); raise it to be stricter. It is the textual-optimization analogue of Koehn's bootstrap significance test for metric differences.

paired is stronger because both sides were scored on the same val tasks, so per-task difficulty cancels and only the paired variance counts; significant treats the two means as independent samples and is only correct when they are.

Single-trial scores report stderr=0, collapsing k·SE to 0 — then significant silently degrades to strict and accepts any positive blip. If you run the significance gate, score with multiple trials (see evaluate).