test-guided-bug-detector

Test-Guided Bug Detector

Analyze failing tests to detect and explain functional bugs in code.

Overview

When tests fail, they provide valuable clues about bugs in the code. This skill analyzes:

1. Test failure output - Error messages, stack traces, assertion failures
2. Test expectations - What the test expects to happen
3. Actual behavior - What actually happened
4. Code execution path - Which code was executed
5. Suspicious patterns - Common bug patterns that match the failure

The goal is to identify the root cause bug and explain why the test exposes it.

Bug Detection Workflow

Failing Test Output
    ↓
Parse Failure Information
    ↓
Identify Test Expectations
    ↓
Trace Execution Path
    ↓
Analyze Discrepancy
    ↓
Identify Suspicious Code
    ↓
Explain Bug Mechanism
    ↓
Suggest Fix

Analysis Process

Step 1: Parse Test Failure

Extract key information from test output:

What to extract:

• Test name and location
• Failure type (assertion, exception, timeout, etc.)
• Expected vs actual values
• Stack trace
• Error messages

Example:

FAILED tests/test_calculator.py::test_divide - AssertionError: assert 0 == 5
Expected: 5
Actual: 0

Stack trace:
  File "tests/test_calculator.py", line 15, in test_divide
    assert divide(10, 2) == 5
  File "src/calculator.py", line 8, in divide
    return a // b

Step 2: Understand Test Intent

Determine what the test is trying to verify:

Questions to answer:

• What functionality is being tested?
• What are the inputs?
• What is the expected output?
• What properties should hold?

Example:

def test_divide():
    # Intent: Verify division returns correct result
    result = divide(10, 2)
    assert result == 5  # Expects 10 / 2 = 5

Step 3: Trace Execution Path

Follow the code path from test to failure:

Trace elements:

• Function calls in stack trace
• Control flow decisions
• Data transformations
• Return values

Example trace:

test_divide()
  → divide(10, 2)
    → return a // b  (integer division)
    → returns 5
  → assert 5 == 5  ✓ Should pass!