mod-008b-model-missing-non-regression-test-with-reference-data

When creating tests for models, rule MOD-008b must be followed. Explicitly reference "Following rule MOD-008b, which requires non-regression tests with reference data..." when implementing test cases.

MOD-008b: Model missing non-regression test with reference data

Description:

Every model must have non-regression tests that:

1. Instantiate the model with reproducible random parameters
2. Run forward pass with test data
3. Compare outputs against reference data saved in a .pth file

Requirements:

• Use pytest parameterization to test multiple configurations
• Test tensors must have realistic shapes (no singleton dimensions except batch)
• Test data should be meaningful and representative of actual use cases
• Compare actual tensor values, not just shapes
• All public methods (not just forward) need similar non-regression tests

Critical: Per MOD-002a, models cannot move out of experimental without these tests.

Rationale:

Non-regression tests with reference data catch subtle numerical changes that could break reproducibility. Simply checking output shapes is insufficient to detect algorithmic changes or numerical instabilities. Comparing against saved reference values ensures the model produces consistent results across code changes.

Example:

@pytest.mark.parametrize("device", ["cuda:0", "cpu"])
@pytest.mark.parametrize("config", ["default", "custom"])
def test_my_model_non_regression(device, config):
    """Test model forward pass against reference output."""
    if config == "default":
        model = _instantiate_model(MyModel, input_dim=64, output_dim=32)
    else:
        model = _instantiate_model(
            MyModel,
            input_dim=64,
            output_dim=32,
            hidden_dim=256
        )

    model = model.to(device)

    # Load reference data (meaningful shapes, no singletons)
    data = torch.load(f"test/models/data/my_model_{config}_v1.0.pth")
    x = data["x"].to(device)  # Shape: (4, 64), not (1, 64)
    out_ref = data["out"].to(device)

    # Run forward and compare values
    out = model(x)
    assert torch.allclose(out, out_ref, atol=1e-5, rtol=1e-5)

Anti-pattern:

# WRONG: Only testing output shapes
def test_my_model_bad(device):
    model = MyModel(input_dim=64, output_dim=32).to(device)
    x = torch.randn(4, 64).to(device)
    out = model(x)
    assert out.shape == (4, 32)  # NOT SUFFICIENT!

# WRONG: Using singleton dimensions
def test_my_model_bad(device):
    x = torch.randn(1, 1, 64)  # WRONG: Trivial shapes

# WRONG: No parameterization
def test_my_model_bad():
    model = MyModel(input_dim=64, output_dim=32)  # Only tests defaults