regression-triage

Regression Triage

Given the regression artifact from a sharded test262 CI run, bucket the regressions by path prefix and error message pattern, sample each concentrated bucket to determine root cause, and classify as false positive, known gap, or real regression with a fix path.

When to use

• After a sharded run fails with N regressions but the net delta is positive (e.g. "+479 pass with 80 regressions")
• When you need to decide which regressions block a baseline commit vs which get filed as follow-ups
• When multiple merges could have introduced overlapping fails and you need to untangle them

Distinct from:

• handle-regression — fixing a single specific regression
• analyze-regression — diffing two JSONL result files (that's step 1 here)

Step 1: Download the merged report artifact

mkdir -p output/regressions-$(date +%s)
gh run download <RUN_ID> -n test262-merged-report -D output/regressions-<TIMESTAMP>

Files: test262-report-merged.json, test262-regressions.txt, test262-results-merged.jsonl.

test262-regressions.txt only has summary stats. Extract the actual regression LIST by diffing the merged jsonl against the current baseline:

Step 2: Compute the regression list

The baseline JSONL is fetched on demand from loopdive/js2wasm-baselines (#1528). Make sure the local cache is populated before diffing:

node scripts/fetch-baseline-jsonl.mjs

import json
base = {}
with open('.test262-cache/test262-current.jsonl') as f:
    for line in f:
        try:
            d = json.loads(line)
            base[d['file']] = d['status']
        except: pass

new = {}
new_details = {}
with open('/tmp/regressions-<TS>/test262-results-merged.jsonl') as f:
    for line in f:
        try:
            d = json.loads(line)
            new[d['file']] = d['status']
            new_details[d['file']] = d
        except: pass

regressions = [(f, new.get(f, 'missing'), new_details.get(f, {}))
               for f, s in base.items()
               if s == 'pass' and new.get(f, 'missing') != 'pass']
print(f"Total: {len(regressions)}")