judging-test-quality

Judging Test Quality

Core principle

A test's value is not "it passes". A test's value is "it would fail if the code were broken in a realistic way". Coverage is not evidence. Green is not evidence. The only evidence is mutation resistance.

When to invoke this skill

• Before writing new tests in an area — know the existing baseline first.
• After a bug ships that the suite "should have caught".
• When a test file has 100% coverage but users still report regressions.
• When onboarding to a module and the suite feels suspicious.

The rubric

For every test, assign exactly one verdict:

Verdict	Criterion
STRONG	Would die under multiple plausible mutations. Asserts exact semantic effects. Tests a contract.
ACCEPTABLE	Would die under at least one plausible mutation. Assertions are specific enough to have meaning.
WEAK	Survives common mutations. Assertions are vague, happy-path-only, or implementation-coupled.
KILL	Actively misleading. Asserts nothing meaningful or only internal mechanics. Delete or rewrite from scratch.

No "borderline" verdicts. When unsure between two, pick the lower.

The mutation checklist

For each test, ask: would this test fail if any of these mutations were applied to the production code?

• > → >=, < → <=, == → !=
• && → ||
• true → false as a literal
• Remove a throw; return null / default instead
• Return empty collection instead of the real result
• Silently swallow an error (catch {})
• Skip a side effect (no reload after save, no selection update after delete)
• Invert a guard
• Off-by-one in a slice or range
• Return the input instead of the computed result
• Reorder two side effects that must run in a specific order

Automatic downgrades

These apply across the runners mutation testing targets — jest and vitest in the JavaScript/TypeScript world, pytest in Python — since the same vacuous assertions show up in every one. A test earns at most WEAK if any of these apply: