test-flakiness

Test Flakiness Detection

A flaky test is one that sometimes passes and sometimes fails without any code change. Flaky tests are worse than no tests in some ways — they train the team to ignore red CI runs, masking genuine failures. This skill identifies them, explains likely causes, and recommends whether to quarantine or fix each one.

Output: Updated tests/regression-suite.md quarantine section + optional production/qa/flakiness-report-[date].md

When to run:

• Polish phase (tests have had many runs; statistical signal is reliable)
• When developers start dismissing CI failures as "probably flaky"
• After /regression-suite identifies quarantined tests that need diagnosis

---

1. Parse Arguments

Modes:

• /test-flakiness [ci-log-path] — analyse a specific CI run log file
• /test-flakiness scan — scan all available CI logs in .github/ or standard log output directories
• /test-flakiness registry — read existing regression-suite.md quarantine section and provide remediation guidance for already-known flaky tests
• No argument — auto-detect: run scan if CI logs are accessible, else registry

---

2. Locate CI Log Data

Option A — GitHub Actions (preferred)

Check for test result artifacts:

ls -t .github/ 2>/dev/null
ls -t test-results/ 2>/dev/null

For Godot projects: GdUnit4 outputs XML results compatible with JUnit format. Check test-results/ for .xml files.

For Unity projects: game-ci test runner outputs NUnit XML to test-results/ by default.

For Unreal projects: automation logs go to Saved/Logs/. Grep for Result: Success and Result: Fail patterns.

Option B — Local log files

If a path argument is provided, read that file directly.

Option C — No log data available

If no logs found:

"No CI log data found. To detect flaky tests, this skill needs test result history from multiple runs. Options:

1. Run the test suite at least 3 times and collect the output logs
2. Check CI pipeline output and save a log to test-results/
3. Run /test-flakiness registry to review tests already flagged as flaky in tests/regression-suite.md"