test-flakiness

Test Flakiness

Locate tests whose result depends on timing, ordering, or hidden state rather than on the system under test. Maintain a registry of known flakes with quarantine status and fix targets.

---

Phase 1: Source CI History

Determine where test history lives. Try in order:

1. .test-history/ directory — JSON or JUnit XML reports stored locally.
2. gh CLI — fetch recent CI runs: gh run list --workflow=test.yml --limit=N --json conclusion,databaseId
3. Bitrise API — if configured, fetch recent builds.
4. CircleCI API — if configured.
5. Manual import — if none of the above, ask the user to drop JUnit XML files into .test-history/ and rerun.

Default window: last 30 days OR last 50 runs, whichever yields more data. Override via --days or --runs.

If history is unreachable, stop and tell the user how to enable it.

---

Phase 2: Parse Results

For each retrieved run, extract:

• Run ID
• Date
• Branch
• Commit SHA
• Per-test result (pass / fail / error / skipped)
• Per-test duration

Aggregate per test:

• Total runs
• Pass count
• Fail count
• Pass rate
• Standard deviation of duration
• First failure date, last failure date

---

Phase 3: Classify

Apply these thresholds:

Class	Criterion
HEALTHY	Pass rate 100% over the window.
FLAKY	Pass rate >= 50% but < 100%, with at least one fail and one pass on the same SHA.
BROKEN	Pass rate < 50% — likely a real regression, not flakiness.
SLOW	Duration std-dev > 2x median, even if pass rate is 100%.
DEAD	Skipped on every run in the window.

Surface FLAKY first; that is the target of this skill.

---

Phase 4: Gather Context for Each Flaky Test

For each FLAKY entry, read the test file and look for:

• Direct timing dependencies — setTimeout, Thread.sleep, Future.delayed, hardcoded animation duration waits.
• Real network or file I/O.
• Order-dependence — module-scope mutable state, missing teardown, reliance on previous test's side effects.
• Real device clock — Date(), DateTime.now(), Date.now() without injection.
• Real randomness — unseeded Math.random, Random().
• Concurrency — uncontrolled await Promise.all(...), async/await without explicit synchronization.
• E2E specific — fixed waits, hardcoded element coordinates, missing retry on flakey selectors.