bisect-regression

Bisect Regression

Find which commit introduced a regression using git bisect with an automated test script.

Prerequisites

• Two archived test262 JSONL files: one "good" (higher pass count) and one "bad" (current/lower)
• Or: a known good commit hash and the current HEAD

Step 1: Identify the regression

# Compare current vs archived results
python3 -c "
import json, os

current = {}
with open('benchmarks/results/test262-results.jsonl') as f:
    for line in f:
        r = json.loads(line)
        current[r['file']] = r['status']

# Find archived files
archives = sorted([f for f in os.listdir('benchmarks/results')
                   if f.startswith('test262-results-') and f.endswith('.jsonl')])
print('Available archives:', archives)

# Compare with most recent archive
if archives:
    prev = {}
    with open(f'benchmarks/results/{archives[-1]}') as f:
        for line in f:
            r = json.loads(line)
            prev[r['file']] = r['status']

    regressions = [t for t in prev if prev[t] == 'pass' and current.get(t) != 'pass']
    fixes = [t for t in current if current[t] == 'pass' and prev.get(t) != 'pass']
    print(f'Regressions: {len(regressions)}, Fixes: {len(fixes)}')

    # Pick a sample regression test for bisecting
    if regressions:
        print(f'Sample test for bisect: {regressions[0]}')
"

Step 2: Run git bisect with vitest filter

GOOD="<commit-hash-where-test-passed>"  # e.g., the baseline commit
BAD="HEAD"

git bisect start
git bisect bad $BAD
git bisect good $GOOD

# vitest exits 0 on pass, 1 on fail — perfect for bisect
git bisect run npx vitest run tests/test262-vitest.test.ts -t "<test-name-pattern>"

Example with a specific test file:

git bisect run npx vitest run tests/test262-vitest.test.ts -t "S7.9.2_A1_T6"

This binary-searches through commits (~7 steps for 90 commits) and reports the first bad commit.

Step 3: Analyze the result

# git bisect will print: "<hash> is the first bad commit"
git show <hash> --stat
git show <hash>