test-validity-checker

Test Validity Checker

Auto-validates that tests are meaningful and catch real bugs.

Activation Triggers

This skill activates when:

• Writing test files
• Before pytest execution
• When test coverage is checked
• When dev loop is running

Validity Checks

Check 1: Empty Test Detection

# INVALID - Empty body
def test_user():
    pass

# INVALID - Only setup, no assertions
def test_create():
    user = create_user()
    # No assertions!

# VALID
def test_user_creation():
    user = create_user()
    assert user.id is not None
    assert user.is_active

Action: Flag empty tests, require assertions

Check 2: Trivial Assertion Detection

# INVALID - Always passes
def test_always_passes():
    assert True

def test_truthy():
    user = create_user()
    assert user  # Just checks existence

# INVALID - Testing constants
def test_constant():
    assert 1 + 1 == 2

# VALID - Tests actual behavior
def test_user_email_lowercase():
    user = create_user(email='[email protected]')
    assert user.email == '[email protected]'

Action: Require value comparisons, not just truthiness

Check 3: Assertion Count

# WEAK - Only 1 assertion
def test_single_assertion():
    response = client.get('/api/users/')
    assert response.status_code == 200

# STRONG - Multiple assertions
def test_list_users():
    response = client.get('/api/users/')
    assert response.status_code == 200
    assert 'results' in response.data
    assert len(response.data['results']) > 0
    assert 'email' in response.data['results'][0]

Minimum: 2 meaningful assertions per test

Check 4: Edge Case Coverage

For each function under test, require:

• Happy path (valid input -> expected output)
• Invalid input (validation error)
• Boundary conditions (min, max, empty)
• Error handling (exceptions caught)

# Complete test suite example
class TestUserService:
    # Happy path
    def test_create_user_success(self):
        ...

    # Invalid input
    def test_create_user_invalid_email(self):
        with pytest.raises(ValidationError):
            ...

    # Boundary
    def test_create_user_max_length_name(self):
        user = create_user(name='x' * 255)  # Max length
        ...

    # Error handling
    def test_create_user_database_error(self, mocker):
        mocker.patch('app.models.User.save', side_effect=DatabaseError)
        with pytest.raises(ServiceError):
            ...