atdd-mutate

Mutation Testing

Add a third validation layer to the ATDD two-stream testing approach. Acceptance tests verify WHAT, unit tests verify HOW, mutation testing verifies that the tests actually catch bugs.

Core Concept

Mutation testing introduces deliberate bugs (mutants) into source code, then runs the test suite. If tests fail, the mutant is killed (good). If tests pass despite the bug, the mutant survives (test gap found).

Source code → introduce mutation → run tests
                                     ├── tests FAIL → mutant killed ✓
                                     └── tests PASS → mutant survived ✗

A project with 100% code coverage can still have a 60% mutation score — meaning 40% of introduced bugs go undetected by the test suite.

When to Use

Run mutation testing after both test streams are green:

1. Acceptance tests pass (WHAT is correct)
2. Unit tests pass (HOW is correct)
3. Mutation testing — verify tests actually detect regressions

This is Phase 6 in the team-based ATDD workflow, or a standalone quality check at any point during development.

Approach: Custom Mutation Tool (Preferred)

The preferred approach is to build a custom mutation tool for the project. This follows the methodology Uncle Bob developed for empire-2025 — a project-specific tool that walks the AST/source tree, applies one mutation at a time, runs targeted tests, and reports survivors.

Why Custom is Preferred

• Tight integration with the project's test runner and source structure
• Targeted execution — run only the tests affected by each mutation
• Language-agnostic — works for any language, including those without established mutation frameworks
• No external dependencies — the tool lives in the project
• AST-level precision — understands the language's constructs natively

Architecture (4 modules)

1. Mutations — rules table (e.g., + → -, true → false, >= → >) plus matching logic that walks the AST/form tree
2. Runner — source-to-test mapping, test execution, pass/fail capture
3. Core — orchestration: read source → discover sites → apply one at a time → run tests → restore original → report
4. Hashing/Selection — hash each function (AST-level) and its covering tests; call dae_mutmap.py to mutate only changed functions and to update the manifest after the run. See the Differential Mutation Testing section.