measurement-discipline

Measurement Discipline

Before accepting a measured claim, first ask under what conditions that measurement holds. Nine principles, each drawn from a failure in practice (the sources are measurements in the author's Contemplative Agent project, shimo4228/contemplative-agent; the principles themselves break the same way in any repo).

1. One success is not evidence

Do not say "it worked" after one smoke run and commit. In a stochastic system (LLMs, external I/O, timing-dependent code), one run is one sample from a distribution. Before making the claim, decide how many runs, under which conditions, would let you say it — if you cannot decide, weaken the claim to "it can work". (Source: the 2026-06 correction, after a single observed run of a follow fix was nearly reported as "fixed")

2. Gate on observations, not the calendar

Do not make the "move to the next stage" condition of a staged or partial rollout a date or a duration. Estimate the required number of observations in advance, and move on when those observations have accumulated. A calendar gate fires even with zero observations; an observation gate stops when there is no data — and stopping is correct. (Source: the enforcement decision for a shadow instrument was cut by the number of verdicts, not by "in two weeks")

How to set the observation count and open the gate:

• Derive n from the pre-registered question (for the precision of a rate, the binomial 95% CI half-width ≈ 1/√n; ±6 pt needs 300). "Once enough has accumulated" is not an n
• Convert n into days using the arrival rate and write it in the ledger (resume condition: 300 answered rows (60–105 rows/day, 3–5 days after the switch)). Measure the arrival rate at every reading and update it as a range — an observation-count condition with no expected date has nothing to check against
• Open the gate on the day n is reached. A weekly ritual is not a clock — a gate that opens only at a ritual makes you wait many times longer than n requires (Source: contemplative-agent relevance shadow; at 60–105 rows a day, a question needing n = 300 had a clock of "4 Saturdays or 1,000 rows". Read 2026-09-26)
• If n is not reached within a short cap (default 14 days), decide instead of extending — retire, or shrink the question to lower n. The longer you wait, the longer it sleeps in the ledger