ai-regression-testing

AI Regression Testing

Testing patterns for AI-assisted development, aimed at a specific failure mode: when the same model writes a fix and then reviews that fix, it carries the same blind spot into both steps. The review says "looks correct" and the bug survives.

AI writes fix -> AI reviews fix -> AI says "looks correct" -> bug still there

This isn't a reason to distrust AI-written code generally — it's a reason to make the check mechanical instead of another round of judgment from the same source. A test either passes or it doesn't; it has no blind spots to share with the author.

The pattern to watch for: parallel-path drift

The single most common AI-introduced regression in codebases that have two code paths returning "the same" data — a sandbox/mock mode next to a real database path, or a feature-flagged variant next to the default — is that a fix lands on one path and not the other. The model reasons about the path it's looking at, patches it correctly, and doesn't notice the sibling path exists unless something forces it to check.

// BAD: sandbox path returns a different field set than production
func GetProfile(w http.ResponseWriter, r *http.Request) {
    if isSandboxMode(r) {
        writeJSON(w, sandboxProfile{ID: "u1", Email: "[email protected]", Name: "A"})
        return // forgot NotificationSettings here
    }
    p := loadProfileFromDB(r.Context())
    writeJSON(w, profile{ID: p.ID, Email: p.Email, Name: p.Name,
        NotificationSettings: p.NotificationSettings})
}

A human reviewing "did the fix work" by hitting the production endpoint will see it work — the sandbox path's drift is invisible unless something tests both paths against the same contract.

Test the contract, not the implementation

Define the response contract once, then assert both paths satisfy it:

package profile_test

import (
    "encoding/json"
    "net/http/httptest"
    "testing"
)

// requiredFields is the contract every code path must satisfy.
// notification_settings was added here after a bug shipped without it.
var requiredFields = []string{
    "id", "email", "name", "notification_settings",
}

func TestGetProfile_AllPaths_SatisfyContract(t *testing.T) {
    cases := []struct {
        name    string
        sandbox bool
    }{
        {"production path", false},
        {"sandbox path", true},
    }

    for _, tc := range cases {
        t.Run(tc.name, func(t *testing.T) {
            rr := httptest.NewRecorder()
            req := newTestRequest(t, tc.sandbox)

            GetProfile(rr, req)

            var body map[string]any
            if err := json.NewDecoder(rr.Body).Decode(&body); err != nil {
                t.Fatalf("decode response: %v", err)
            }
            for _, field := range requiredFields {
                if _, ok := body[field]; !ok {
                    t.Errorf("%s: missing required field %q", tc.name, field)
                }
            }
        })
    }
}