fixing-flaky-tests

fixing-flaky-tests is a skill for Claude Code, Codex from amplifthq/opentag. It costs 220 tokens per session (4,919 once invoked), scanned A, a copy of fixing-flaky-tests, MIT.

A workflow for reproducing, diagnosing, fixing, and validating flaky tests in the PostHog monorepo. A flaky test passes and fails unpredictably, while a monorepo stores multiple projects in one repository.

In plain words
What is it for?
Investigating intermittent CI failures, finding their root cause, deciding whether a test should move or be removed, and validating the result over multiple runs.
Why use it?
It avoids hiding intermittent failures with sleeps, retries, or longer timeouts and requires evidence from repeated runs before treating a problem as fixed.

Skill for Claude CodeCodex

Written for no agent in particular: nothing here depends on one. Also seen: installed under .agents/ (shared by several agents).

Good fit Investigating intermittent CI failures, finding their root cause, deciding whether a test should move or be removed, and validating the result over multiple runs.

Compare 6 skills from other repositories ↓
Install with agentmods
npx agentmods add skills/amplifthq/opentag/fixing-flaky-tests
About the project

OpenTag connects Slack with coding agents running on a computer controlled by the user, so channel mentions can queue work and return status, decisions, and evidence in the same thread. It is for engineering teams that want to run agents such as Claude Code, Codex, or Cursor against local checkouts through the Agent Client Protocol.

amplifthq/opentag · 0 stars · on GitHub · openmeld.ai

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

Any agent
npx skills add amplifthq/opentag --skill fixing-flaky-tests
Clone the repo
git clone --depth 1 https://github.com/amplifthq/opentag

Made for: Claude Code, Codex.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

agentmods badge for fixing-flaky-tests

README.md
[![agentmods](https://agentmods.dev/badge/skills/amplifthq/opentag/fixing-flaky-tests/github.svg)](https://agentmods.dev/skills/amplifthq/opentag/fixing-flaky-tests)
Your own site
<a href="https://agentmods.dev/skills/amplifthq/opentag/fixing-flaky-tests"><img src="https://agentmods.dev/badge/skills/amplifthq/opentag/fixing-flaky-tests/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

agentmods 80×15 button for fixing-flaky-tests

Your own site · 80×15
<a href="https://agentmods.dev/skills/amplifthq/opentag/fixing-flaky-tests"><img src="https://agentmods.dev/badge/skills/amplifthq/opentag/fixing-flaky-tests.svg" alt="Reviewed on agentmods" width="80" height="20"></a>
Per session 220 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.
When invoked 4,919 The whole file, excluding the scripts and references it only reads on demand.
Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe.
Origin 100% copy Near-identical to another mod in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5.1 $0.00220 $0.04919
Opus 5.5 $0.00088 $0.01968
Sonnet 5.5 $0.00044 $0.00984
Haiku 4.5 $0.00022 $0.00492

Measured 2d ago against content hash 7dfddbf3bbd0, method: parsed. Prices are Anthropic first-party input rates as of 2026-10-07, from the pricing page.

Security

Grade A, and why

fixing-flaky-tests scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

Origin

This is a copy

100% identical to fixing-flaky-tests — 33 lines differ, which has more behind it and is treated as the original. This page carries a canonical link to it rather than competing with it.

.agents/skills/fixing-flaky-tests/SKILL.md · 298 lines

How it starts

The opening of the file, as written. The whole thing — 298 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Fixing flaky tests

Before you propose a change to how the suite runs in CI, check things already tried. It records measured verdicts on test parallelism, sharding, and coverage-based selection, so a rejected approach is not rebuilt.

Three non-negotiables, in order:

  1. Reproduce before you fix. A fix for a failure you never observed is a guess. Only fall back to analytical fixes when the escalation ladder below is exhausted.
  2. Fix the root cause. Sleeps, raised timeouts, retries, and weakened assertions hide flakes; they do not fix them.
  3. Validate with an N-run loop. One green run proves nothing about an intermittent failure. Size N to the observed failure rate.

Stabilizing the test is not the only valid ending. Once you know why it flakes, step 5 asks whether it should exist. A test that catches nothing real is worth deleting, and one that flakes because of the level it runs at is worth moving down a rung.

Before any of these: measure, don't assume. Flaky-vs-deterministic, and the rate, are facts to establish from verifiable GitHub run data (step 1) — never inherited from a Slack alert, a teammate's guess, or a ci:insights label.

For triaging a red CI run (finding and classifying the failure), use the debugging-ci-failures skill first — this skill takes over once the failure is classified as a flaky test. For writing new Playwright tests that aren't flaky, use the playwright-test skill. investigating-ci-failures (green/red boundary) and diagnosing-ci-and-merge-bottlenecks (the engineering-analytics-flaky-tests tool's caveats) are product skills under products/engineering_analytics/skills/, not invocable here: read their SKILL.md at that path.

1. Measure the failure rate — from GitHub, not from a digest

The GitHub Actions API (or GitHub MCP) is the source of truth. hogli ci:insights is a digest, not an oracle — it can mislabel flaky-vs-deterministic, it lags the API until GitHub's webhook settles, and it cannot give you a rate at all: it reports absolute counts, because CI emits failures but omits ordinary passing runs, so there is no denominator. Use it to validate a hypothesis or pull historical context, never as the first move or the classification authority.

Read the full file on GitHub · 298 lines

Files

What ships with it

2 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 2d ago First seen · 298 lines · 220 tokens per session scan A 7dfddbf3bbd0

Subscribe to this mod's changes

fixing-flaky-tests is a skill published in the GitHub repository amplifthq/opentag (0 stars, last pushed 2d ago), licensed MIT. It adds 220 tokens to every session and 4,919 once invoked, about $0.0009 per session on Opus 5.5. A static security scan graded it A with 0 findings. It is 100% identical to fixing-flaky-tests, differing in 33 lines, and is treated as a copy.