test-case-reducer

Test Case Reducer

Automatically reduce bug-triggering test cases to minimal form while guaranteeing the reduced test still reproduces the same failure.

Core Concept

Test case reduction works by:

1. Identify: Parse the test case and identify reducible elements
2. Remove: Systematically try removing elements (lines, statements, inputs)
3. Validate: Run the test after each removal to verify failure is preserved
4. Iterate: Continue until no further reduction is possible

Workflow

1. Prepare the Test Case

Ensure you have:

• A test case that consistently reproduces the failure
• A way to run the test (command, script, test framework)
• A clear failure condition (oracle)

2. Define the Oracle

The oracle determines if the test still exhibits the expected failure:

Exit code:

• Test fails with specific exit code (e.g., non-zero)
• Example: --oracle exit_code --expected 1

Exception type:

• Test raises specific exception
• Example: --oracle exception --expected ValueError

Output pattern:

• Test output contains specific text
• Example: --oracle output_pattern --expected "division by zero"

Assertion:

• Test fails on specific assertion
• Example: --oracle assertion --expected AssertionError

3. Choose Reduction Strategy

Aggressive (fastest, smallest result):

• Binary search reduction first
• Delta debugging second
• Greedy line-by-line cleanup
• Use when: Test execution is fast, minimal size is critical

Balanced (recommended):

• Delta debugging only
• Good balance of speed and quality
• Use when: Standard reduction needs

Conservative (safest, most readable):

• Greedy line-by-line only
• Preserves more context
• Use when: Readability matters, test is already small

4. Run the Reduction

Using the provided script:

python scripts/reduce_test.py test_file.py \
  --command "python test_file.py" \
  --oracle exit_code \
  --expected 1 \
  --strategy balanced

Or manually apply reduction algorithms (see references/algorithms.md).