security-regression-tests

Security Regression Tests

Rules (for AI agents)

ALWAYS

• Prove the test fails without the fix. Revert the patch, or mutate the guard, and confirm the new test goes red — then restore. A test written after the fix can be vacuously green: wrong route, wrong fixture user, wrong assertion target, a typo in the payload. It will stay green forever and nobody will learn anything from it. This is the one step that distinguishes a guard from a decoration, and it is the one most often skipped.
• Assert both directions: the attack input now yields the secure outcome, and a legitimate input still succeeds. A deny-only test passes equally well against an endpoint that is broken for everyone, so it will hold the line and tell you nothing when someone disables the feature entirely.
• Assert the effect, not only the status code. A 403 proves the request was refused; it does not prove the write did not land somewhere else, that the record was not returned in a different shape, or that a queue message was not emitted. Check the state the attack was trying to reach.
• Seed the fixtures the assertion depends on before writing it. A cross-tenant test needs two principals and two owned resources, deterministically created — if both test users end up in the same tenant, the deny assertion is untestable and passes for the wrong reason.
• Make it deterministic and offline. Freeze the clock for expiry and TTL assertions, stub the HTTP client, dialer or resolver rather than reaching the network, fix random seeds, and avoid sleep-based timing assertions and state shared between tests. Note that a canonical SSRF payload such as http://169.254.169.254/latest/meta-data/ is a live, reachable address on a cloud CI runner, so the naive test both leaves the sandbox and returns a different verdict on a laptop.
• Graduate a live proof-of-concept by keeping its shape and dropping its mechanism. dynamic-verification confirms some classes with a timing delta or an out-of-band callback, and neither survives in CI: replace the time-based payload with an assertion that the query was parameterized, and the callback with a stubbed dialer asserting no outbound request was attempted.
• Put the test in a job that can block the merge. A committed test in a job that is continue-on-error, not a required check, or nightly-only is documentation. cicd-security owns the gate configuration itself.
• Name it so a reviewer sees what it guards — test_idor_orders_cross_tenant_403 — and reference the finding in the test body or the commit, so the next person to see it fail knows what it is protecting rather than what it is blocking.