validate-your-validator

Validate your validator

A checker that cannot fail is not checking anything. It is a green light wired to nothing, and it is worse than no checker at all, because now everyone has stopped looking.

The test is simple and non-negotiable: plant the failure it exists to catch, and confirm it catches it. Until you have watched it go red on purpose, a green result carries no information.

When to use this

• You are writing a test, a lint rule, a scanner, a validator, a probe
• You are adding a health check or an alert
• You are about to trust a green CI run on something important
• A checker has been green for a suspiciously long time
• Someone says "the scan found nothing"

The procedure

1. Write the checker.

2. Plant a failure. Introduce, in a fixture or a scratch copy, exactly the thing the checker exists to detect. A real one, not a simplified one.

3. Run it. It must go red. If it does not, the checker is the bug — fix it before you fix anything else.

4. Plant a control: something you know is fine. Run it. It must go green. A checker that flags everything is as useless as one that flags nothing, and it is more expensive, because people learn to ignore it and then ignore the true positive too.

5. Keep both as permanent fixtures inside the checker. This is the part everyone skips. The planted failure and the control become cases the checker runs against itself, every time. The day someone refactors the pattern and it stops detecting anything, the suite goes red instead of going quiet.

6. Write down what it cannot catch. Every checker has a blind spot. An unwritten blind spot gets read as full coverage by the next person.

The case that earns this skill its place

A secret-detection step was added to a CI pipeline. It incriminated itself three times before it was fit to run.

First, it searched for bare credential prefixes. Those prefixes were written in its own rule list, in its own source file, which the scan walked. It reported itself as a leak. Excluding its own file made it green — and would have made it green forever, because there was no case proving it could still detect a real one.

Second, the sentinel string used as its own control appeared verbatim in the file. So the "proof it works" and the "thing it found" were the same bytes. It was detecting its own test data and reporting a pass.

Third, and worst, it flagged TOKEN = os.environ.get("X") as a hardcoded secret. That line is the correct pattern — it is a secret being read from the environment, the exact thing you want people to write. The checker was punishing the good behaviour and, by symmetry, had never been shown a real hardcoded value to see whether it would fire on one.

What fixed it was not a better pattern. It was two fixtures: a file containing a genuine-shaped credential that the suite asserts is flagged, and a file containing environment reads and placeholders that the suite asserts is not. Both live next to the scanner. Break the scanner now and the suite tells you.