robust-testing

Robust Testing & Verification Invariants

Guidelines for designing deterministic verification boundaries ($V(\mathbf{S})$). This skill instructs the agent on how to move beyond simple input-output example tests to construct property-based, metamorphic, and fuzz-testing harnesses that prevent regression and self-deception in generated code.

---

1. Core Philosophy

Traditional unit testing asserts that specific hardcoded inputs produce specific outputs. For AI-generated code, this example-based approach creates a self-deception loop: the agent generates both the implementation and the tests, propagating the same logical blind spots to both files.

To establish a mathematically sound validation boundary, you must verify the code against properties and invariants rather than static examples. A program is verified when its state transition satisfies high-level algebraic equations across a randomized input space.

Iterative Refinement of Verification Boundaries ("Testing the Tests")

Because there is no deterministic validator to "test the test," the test suite must be converged and refined with the same rigor as the implementation:

• Specification Traceability: Every test case must explicitly trace to a constraint in the specification or state-space model.
• Non-Trivial Failure Validation: The baseline check is absolute: a test suite that passes on empty or unimplemented code ($\Delta E_0 = 0$) is invalid.
• Input Domain Scrutiny: Generators must be audited to verify they generate the complete phase-space of input variables, explicitly including edge states (null, negative, empty structures, maximum limits).

---

2. Selection & Execution Rules

When executing the closed-loop optimization cycle, select verification methods based on the task domain:

• Formal Specification Active: Tests MUST trace directly to specification invariants, achieving 100% assertion coverage of normative constraints.
• Algebraic Invariants Present: If the domain exhibits algebraic properties (isomorphism, commutativity, etc.) and the target language has mature PBT support, Property-Based Testing (PBT) MUST be the primary validation gate.
• Untrusted or Complex Input Boundaries: If the program parses custom protocols, processes serialization formats, or handles external network payloads, Fuzz Testing MUST be applied to stress-test the boundary.
• Oracle-Less Systems: If the expected outputs are complex or computationally expensive to calculate (e.g., optimization routines, matrix operations), Metamorphic Testing MUST be used to assert functional relations across perturbed inputs.