testing

Testing Skill

Guidelines for writing effective tests that actually verify correctness.

Critical Anti-Pattern: Success-Only Tests

Never write tests that only check for successful execution without verifying output.

Bad Example (YAML test suite bug)

// BAD: Only checks parse success, not correctness
#[test]
fn test_sequence_of_mappings() {
    let yaml = b"- name: Alice\n  age: 30";
    let result = YamlIndex::build(yaml);
    assert!(result.is_ok());  // Parser could return completely wrong structure!
}

This test passed even when the parser produced ["name", "Alice", "age", 30] instead of [{"name": "Alice", "age": 30}].

Good Example

// GOOD: Verifies actual output matches expected
#[test]
fn test_sequence_of_mappings() {
    let yaml = b"- name: Alice\n  age: 30";
    let index = YamlIndex::build(yaml).unwrap();
    let json = index.to_json(yaml);
    assert_eq!(json, r#"[{"name":"Alice","age":30}]"#);
}

Test Generator Pitfalls

When generating tests from external sources (like official test suites):

1. Don't Skip Cases Based on Content

# BAD: Skips tests containing quotes - misses most real-world cases
if '"' not in expected_output:
    generate_comparison_test()
else:
    generate_parse_only_test()  # No output verification!

# GOOD: Properly escape and compare all cases
json_escaped = expected.replace('\\', '\\\\').replace('"', '\\"')
generate_comparison_test(json_escaped)

2. Count Actual Comparisons

After generating tests, verify how many actually compare output:

grep -c "assert_eq!" tests/generated_suite.rs
grep -c "is_ok()" tests/generated_suite.rs

If is_ok() count >> assert_eq! count, tests aren't verifying correctness.

3. Benchmark Generators Are Test Coverage Too

A shape that no generator emits is invisible to the entire benchmark suite, so "all benchmarks neutral" is not evidence about it. In #106 every YAML generator emitted - (dash-space), so block-sequence items written as a bare - on its own line had zero coverage — and hid a quadratic path worth up to 16x. The omission was justified by a stale comment claiming the parser required dash-space, which was never true.