ai-agents testing.instructions.md

Test File Rules

Tests under tests/, skill tests/ directories, and .agents/security/benchmarks/ enforce correctness and catch regressions. They are not decoration.

MUST

1. Structural validation. Quality-gate prompt tests MUST comply with ADR-023 structural requirements. Other tests follow the placement and coverage rules below.
2. Pester version. PowerShell tests MUST target Pester 5.7.1+. Python tests MUST target pytest 8+.
3. Evidence for fixture changes. MUST NOT modify baseline fixtures to make failing tests pass. A baseline change MUST cite the behavior change that justifies it.
4. Coverage targets. Coverage MUST meet category minimums (AGENTS.md): 100% security, 80% business, 60% docs.
5. Independent tests. Each test MUST pass in isolation. Shared mutable state is prohibited.
6. Placement. New tests MUST live in tests/ or .agents/security/benchmarks/. Do not add tests to shipped skill directories.
7. A mutation harness MUST report three outcomes, not two. DEAD, SURVIVED, and DID-NOT-APPLY. The conventional reading of a mutation result is that a survivor means a weak test. That reading is unsafe here, because the common cause of a survivor in this repository is that the mutation never applied: the target literal had moved or been reworded, the patch step wrote nothing, the file stayed byte-identical, and the harness then graded unmutated code and printed SURVIVED. A harness that cannot tell those apart reports the same string for "your test is weak" and "I tested nothing." Each mutant MUST count occurrences of its target before patching and refuse an ambiguous or absent match, MUST exit non-zero from the patch step when the literal is missing, MUST cmp -s the patched file against a backup and fail when they match, and MUST assert a byte-identical restore afterward. Measured: one harness reported four survivors of which three had silently failed to apply, and a second reported a clean sweep it had never performed.
8. Assert on the process exit code, not on a helper's return value. A test for a CLI or a gate MUST drive main(argv) (or the documented entrypoint) and assert on the integer it returns or the process status it produces. Asserting that a helper returned a findings list, an error string, or a falsy sentinel proves the helper detected the problem and proves nothing about whether the program fails. This is the dominant silent-pass shape in this repository (Issue #4068, six confirmed instances): a run: block under set -e fails on any non-zero exit, the natural Python translation returns a sentinel instead, no caller converts it, and the step goes green on a failure that used to be red. Every affected case had passing tests, and every one of those tests asserted on a return value.
9. Assert on parsed structure, never on a substring of a structured file. A test that checks wiring in YAML, JSON, or TOML MUST parse the document and assert against the resulting object graph. assert "some-step-name" in workflow_text passes when the step has been deleted and the name survives in a comment, an adjacent job, or an unrelated key, and a fixed-width text window around a match reaches into the neighbouring block. Measured: a wiring test passed after a mutant removed the step's - name: line, and a ratchet-wiring test passed with both ratchets commented out.
10. A scope measurement MUST be reported together with the size of the scope. When a change narrows what a checker examines, the count of findings alone cannot distinguish "nothing is wrong" from "nothing is being checked", because the filter under evaluation is also the filter applied to the evidence. State how many candidates were examined next to how many failed, and MUST NOT cite a zero-finding result whose examined count is unknown. Measured: a configuration change was accepted on a Summary: 0 issues line while Linting: 0 files sat directly above it; the same tool on the same tree reported 823 issues across 141 files once the scope was restored.
11. A mutation harness MUST include at least one inverted control: a mutation the suite is required to SURVIVE, asserted as rc == 0. A harness whose mutants are all the same polarity, apply a change and expect failure, cannot distinguish "every mutant died" from "the harness fails no matter what." Any defect that makes the run fail unconditionally reads as a clean sweep. Measured: a harness passed pytest a dotted module name as a positional argument, pytest read it as a filesystem path, every invocation exited 4 with file or directory not found having collected zero tests, and four mutants asserting only returncode != 0 each reported a kill; the harness graded a suite it had never once run. Adding one mutant that must survive surfaced it on the first execution. A harness that shells out to a test runner MUST also reject the runner's silent-nothing-ran signals directly, at minimum "no tests ran" absent from the output and a returncode that is not pytest's usage-error 4, and MUST target test files by filesystem path rather than by dotted module name.
12. When a change alters concurrency or synchronization semantics, one green run is not evidence. Stress-run it. The conventional reading of a passing suite is that the change is safe. That reading does not hold here, because a single run samples one interleaving and the suite reports it as though it had sampled the space. The trigger is any change to how work is scheduled or serialized; introducing or resizing a thread or process pool, moving work onto an event loop or a queue, adding or removing a lock, and fanning out through asyncio.gather, Promise.all, or goroutines are examples rather than the whole list. Such a change MUST have the affected tests run at least thirty times in a row, and any failure in that window MUST be treated as a defect in the test or in the code rather than as noise. Thirty is not arbitrary: n runs miss a defect that fails on a fraction p of interleavings with probability (1-p)^n, so twelve runs miss a one-in-ten flake 28 percent of the time while thirty miss it 4 percent. That bound assumes the runs are independent and p is stable; reusing a seed, a warm process, or fixed scheduler conditions correlates the interleavings sampled, so the real miss rate runs higher than (1-p)^n and detection falls short. Vary the seed and scheduler conditions, and prefer a fresh process per run where the interleaving is the thing under test. When thirty runs of a whole file cost too much, narrow the selection to the affected class or test and keep the count, because sample size is what buys detection and scope is what costs time.
13. When a first fix does not clear a concurrency failure, record which assertion fails before proposing a second cause. A hypothesis that is plausible and also true can still be insufficient, and only the failure text separates the two. Measured: merging a ThreadPoolExecutor into scripts/maintenance/gc_worktrees.py left the full 85-test set green on the first run and then produced seven failures in twelve under stress; adding a lock to an unsynchronised read-modify-write in the test's fake clock was a real defect and a real fix and still left eleven failures in twenty; reading the failure text showed one test failing on identity and never on cardinality, and re-pinning that one assertion gave twenty-five clean runs in twenty-five. The repair generalises to what a pool actually invalidates, which is order and not identity. Assert the expected set, or the multiset where duplicates are part of the contract, together with the per-item invariant. Fall back to cardinality alone only where identity is itself nondeterministic, because cardinality plus a generic invariant accepts two copies of one item where one of each was required. Worked example in PR #4294.