ci-quarantine

/ci-quarantine

One in four merge-queue builds fails, and most of those failures are not bugs. A build that dies on a flake costs about thirty minutes of queue time for every PR behind it. The quarantine lane takes one named test out of the blocking path while it is being fixed.

A quarantined test still runs, still retries and is still reported. It only loses the power to fail the queue. Nothing is skipped, deleted or filtered out, so the day it stops flaking is visible in the same reports.

It is debt, not a fix. Every entry expires after 7 days, at most 10 can exist at once, and adding one writes a proposed ledger entry that says to fix or delete the test.

Look before you add

git fetch --quiet origin ci-steward-data || true
pnpm ci:flaky

It lists every test that failed and then passed on the same merge-group tree, with counts and dates, and says which qualify. Read the status column before anything else:

• QUALIFIES — flaked on at least two different trees, recently enough that nothing suggests it was fixed. This is the only thing that may be added.
• COOLING — it flaked often and has now been quiet for longer than its own gap between flakes. That is the shape of a test somebody fixed; quarantining it spends a slot on nothing. The add path refuses it.
• BELOW-THRESHOLD — one bad runner. Not a classification.

Right now, --fetch is the only path that returns anything. pnpm ci:flaky reads per-test flake names out of the daily collector's snapshots, and the collector only starts recording them with this change — so until it has run for about a week, the plain command correctly reports nothing. --fetch reads the Actions artifacts directly instead (a few hundred API requests, several minutes); artifacts expire after seven days, so that window is what it can see.

--json saves the classification, and quarantine add --evidence <that file> reuses it instead of downloading everything twice.

Copy the file and title, do not retype them. They are matched byte for byte, so a title that differs by one character — or by Unicode normal form — silently matches nothing. That fails safe (the entry excuses nothing, and the fan-in names it as a quarantined test that did not run), but it wastes a slot.