detect-flaky-tests

Detect Flaky Tests

A test is flaky when it produces both PASS and FAIL outcomes across multiple independent CI runs in the last 7 days, with no code change to that test's package explaining the inconsistency. Cross-PR analysis provides the strongest signal: if the same test fails on PR-A but passes on PR-B, that inconsistency is almost certainly non-determinism, not a legitimate regression.

Flakiness threshold (keeps false-positive rate low)

A test is flagged only when all three conditions hold in the 7-day window:

Condition	Rationale
fail_count >= 2	One failure could be infra noise
pass_count >= 2	One pass could be a pre-fix lucky run
0.05 < fail_rate < 0.95	Outside this band it is either reliably broken or reliably passing

---

Step 1 — Collect workflow run IDs (last 7 days)

SINCE=$(date -u -v-7d +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ)
gh api --paginate \
  "repos/agent-substrate/substrate/actions/workflows/pr-workflow.yaml/runs?status=completed&per_page=100&created=>=$SINCE" \
  --jq '.workflow_runs[] | {id: .id, conclusion: .conclusion, head_sha: .head_sha, created_at: .created_at}'

--paginate is required: a typical week has several hundred completed runs (600+ as of August 2026), far more than one page of 100.

Collect all run IDs. Process both successful and failed runs — both contain test output.

---

Step 2 — Download and parse logs: two jobs, three lanes per run

For each run, you need logs from two jobs:

Job name	Coverage
run-tests	Unit + integration tests (go test -race -v ./...)
e2e-test	E2E suite, both sandbox classes — two sequential steps in the one job

# List all jobs for a run
gh api "repos/agent-substrate/substrate/actions/runs/<RUN_ID>/jobs" \
  --jq '.jobs[] | {id: .id, name: .name, conclusion: .conclusion}'

# Download log for a specific job
gh api "repos/agent-substrate/substrate/actions/jobs/<JOB_ID>/logs" > /tmp/job_<JOB_ID>.log