a-check-that-can-only-confirm

A Check That Can Only Confirm

Contract

Iron law: a check that cannot fail is not a check — prove it can fire, prove it can stay silent, and prove it reached its input. Applies whenever a guard, test, lint rule, roster walk or scan is written or changed. Returns, per check: a must-fire mutant, a must-stay-silent mutant built from a correct input that resembles the incorrect one, an input-removal mutant, an asserted non-zero count of what was examined, and one run against a real full-size input — each one executed, not just written. Permitted skips: none; a check shipped without the three mutants is unverified.

Why a green result proves less than it looks

A guard has two failure modes and only one is loud. Reporting the wrong answer gets noticed. Reaching nothing does not: zero offending files and zero files examined produce the same green. That happens when the matcher is narrower than intended — matching a number only when a unit follows it, handling two of the four spellings a symbol is imported under — so the input the guard exists for never enters it, and nothing in the output says so.

Write three mutants, not one

• Must-fire — an input the check exists to reject. This is the one everyone writes, and on its own it is satisfied by a matcher that fires on almost anything.
• Must-stay-silent — a correct input built to resemble the incorrect one as closely as you can. Without it, over-matching stays invisible until the check blocks real work, and the fix under that pressure is to exclude one more case, which just moves the false positive.
• Input-removal — delete an entry from the roster, empty the directory it walks, rename the symbol it starts from. The check must now fail. This is the mutant that catches a narrowed matcher, because matching nothing and having nothing to match are indistinguishable from the outside.

Make silence say something

Assert and print a count of what the check examined, beside the count of what it found: assert examined > 0 before assert not offending. A run reporting "examined 0 entries, 0 problems" is reporting a defect in itself; without the first number it reads as a pass.