What is flaky-test-audit?

A test-health skill that repeatedly runs tests and calculates how often each one passes or fails. A flaky test is one that passes and fails on the same code without a code change.

Use it to investigate occasional test failures, measure per-test flakiness, audit a test suite on a schedule, and identify tests that need attention.

It replaces guesses about intermittent failures with measured results and can quarantine tests that behave nondeterministically.

flaky-test-audit — Skill by raisedadead

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

agentmods

npx agentmods add skills/raisedadead/claude-code-plugins/flaky-test-audit

Any agent

npx skills add raisedadead/claude-code-plugins --skill flaky-test-audit

Clone the repo

git clone --depth 1 https://github.com/raisedadead/claude-code-plugins

Made for: Claude Code, Codex.

Or install whetstone, the plugin that ships this one along with the rest of its 6 skills, 1 agent.

Per session 66 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 891 The whole file, excluding the scripts and references it only reads on demand.

Security scan A 0 findings. Scan, not verified.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5	$0.00066	$0.00891
Opus 5	$0.00033	$0.00445
Sonnet 5	$0.00013	$0.00178
Haiku 4.5	$0.00007	$0.00089

Measured 2d ago against content hash dd71761ce3df, method: parsed. Prices are Anthropic first-party input rates as of 2026-08-30, from the pricing page.

Security

Grade A, and why

flaky-test-audit scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.

The scan reads SKILL.md. This mod also ships 2 executable files (scripts/compute_flakiness.py, scripts/flake_runner.sh), listed below but not scanned — reading those needs a real analyzer, not pattern matching.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

plugins/whetstone/skills/flaky-test-audit/SKILL.md · 52 lines

How it starts

The opening of the file, as written. The whole thing — 52 lines — stays where its author put it; the contents beside it link to each section on GitHub.

flaky-test-audit — measure flakiness, don't guess it

A test is flaky when it passes and fails on the same code. That is a number: run it N times, count the failures, and any test with a rate strictly between 0 and 1 is flaky by definition. The flag itself is computation, not judgement.

When to use

A test "sometimes fails" and you want it confirmed and quantified.
Periodic test-health sweep (nightly / weekly) that escalates only when a new test turns flaky.

Process

Detect the framework and pick a per-test invocation — see reference/framework-adapters.md. The unit of work is a results.json mapping test -> {"runs", "fails"}.
Run repeatedly. For a single suspect test, or a loop over test ids:
```
FLAKE_TEST_NAME="<test-id>" \
  "${CLAUDE_PLUGIN_ROOT}"/skills/flaky-test-audit/scripts/flake_runner.sh 10 results.json <test-command>
```
Pass the test id verbatim — the runner JSON-escapes it, so an id carrying quotes or backslashes still keys valid JSON. Or produce results.json directly from a framework's JSON reporter aggregated across N runs (adapters doc).
Compute the rate:
```
python3 "${CLAUDE_PLUGIN_ROOT}"/skills/flaky-test-audit/scripts/compute_flakiness.py \
  results.json prev-quarantine.json quarantine.json
```
It prints a sorted FLAKY fails/runs rate test table, writes quarantine.json for every 0 < rate < 1 test, and exits with the count of tests newly flaky since prev-quarantine.json, capped at 250. Exit 1-250 = a scheduled routine escalates; 0 = no new flake, stay quiet.

The codes above the cap are tool failures rather than counts: 251 a bad invocation, 252 a results.json or baseline file that is missing, unparseable or not shaped {test: {runs, fails}}, 253 a quarantine.json that could not be written. 251 and 252 write no quarantine.json, so a sweep that never parsed cannot be read as a clean one. A prev-quarantine.json that does not exist yet is the first sweep, not an error.

Read the full file on GitHub · 52 lines

Files

What ships with it

3 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

2d ago First seen · 52 lines · 66 tokens per session scan A dd71761ce3df

Subscribe to this mod's changes

flaky-test-audit is a skill published in the GitHub repository raisedadead/claude-code-plugins (2 stars, last pushed 7d ago), licensed ISC. It adds 66 tokens to every session and 891 once invoked, about $0.0003 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.

Testing

systematic-debugging

Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes.

obra/superpowers · 21 tokens

brainstorming

You MUST use this before any creative work - creating features, building components, adding functionality, or modifying behavior. Explores user intent, requirements and design before implementation.

obra/superpowers · 37 tokens

auto-perf-optimize

Run agent-driven VS Code performance or memory investigations. Use when asked to launch Code OSS, automate a VS Code scenario, run the Chat memory smoke runner, capture renderer heap snapshots, take workflow screenshots, compare run summaries, or drive a repeatable scenario before heap-snapshot analysis.

microsoft/vscode · 62 tokens

chat-perf

Run chat perf benchmarks and memory leak checks against the local dev build or any published VS Code version. Use when investigating chat rendering regressions, validating perf-sensitive changes to chat UI, or checking for memory leaks in the chat response pipeline.

microsoft/vscode · 51 tokens

chat-pet-sprite-creation

Use when creating or changing VS Code chat pet sprite art, sprite sheets, state animations, eye treatments, Stable/Insiders variants, or pet transitions under src/vs/workbench/contrib/chat/browser/widget/media/chatPet.

microsoft/vscode · 53 tokens

cpu-profile-analysis

Analyze V8/Chrome CPU profiles (.cpuprofile) and DevTools trace files (Trace-.json). Use when: profiling performance, investigating slow functions, comparing code paths, finding bottlenecks, analyzing timeToRequest, understanding call trees from sampling profiler data, analyzing layout/paint/rendering, investigating…

microsoft/vscode · 71 tokens