writing-tests

Writing Tests

A test earns its place by going red the moment a real defect returns — not by executing lines for a coverage number. Every rule below serves that.

Test behavior, not source text

Never grep the implementation as a proxy for what it does. assert "chmod 0444" in script passes whether or not that line ever runs, breaks on a harmless rename, and stays green against a broken implementation that still contains the token. Instead drive the real code path and assert the observable outcome: run the function or script with the real dependency where one can run (a real git in a scratch repo, a real file tree, the real parser, a real CLI against a local server), or where it genuinely cannot, under a stub that records how it was called (a fake install, docker, id on PATH) — and check what actually happened: the file written and its mode/owner, the argv+env a binary was exec'd with, the exit code, the bytes on stdout/stderr, the state left on disk.

The only legitimate exceptions parse data, not imperative logic: validating a config artifact the product ships and loads at runtime (a settings.json, an execpolicy.rules) with its real parser, or pinning a cross-file structural contract two files must agree on when no single process can observe both. A grep of imperative source to confirm it "does X" is never an exception.

Fake the SERVICE, never the tool — and never invent its data

A change whose correctness depends on how an external dependency BEHAVES ships an end-to-end test that drives its whole path against that real dependency — write it FIRST, and let stubs fill only what it cannot reach. The trigger is a dependency whose behavior you would otherwise assert from belief: a CLI, a service, a kernel or filesystem semantic, a parser you do not own. Three kinds of change owe none. A change to pure logic — its unit test already drives the whole path, and there is no far side to be wrong about. A refactor that keeps behavior — the tests it must keep green ARE the end-to-end test, and a new one only re-pins what they pin. A fault branch reachable only by a failure you cannot induce for real (a chmod that root ignores, a vanished path, a tool absent from PATH) — inducing it is what a stub is legitimately for, and there the stub PRODUCES the fault; it never stands in for the tool's ordinary work. A stub answers with whatever the test author believed the real dependency returns, so when the bug under test IS that belief — the commonest kind in code that talks to an external API — the fixture encodes the wrong belief on both sides and the suite goes green through the defect.