Borrowing it
Nothing to install: this file belongs to PostHog/posthog-foss. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.
curl -O https://raw.githubusercontent.com/PostHog/posthog-foss/master/.agents/skills/fixing-flaky-tests/SKILL.mdgit clone --depth 1 https://github.com/PostHog/posthog-fossWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/posthog/posthog-foss/fixing-flaky-tests)<a href="https://agentmods.dev/skills/posthog/posthog-foss/fixing-flaky-tests"><img src="https://agentmods.dev/badge/skills/posthog/posthog-foss/fixing-flaky-tests.svg" alt="Measured on agentmods" height="20"></a>- NVIDIA SkillSpector warn
SkillSpector: 1 finding, up to medium
These are SkillSpector’s own severities. On a checked sample its high-severity flags on skills were ~96% false positives — a documented command, a public API, a “never do X” rule — so we show them as a caution to read, not a verdict. Why →
- medium Prompt Injection · line 247 Large whitespace padding was detected (a block of blank lines or a long run of spaces). This can push injected instructions below or to the right of the visible area so a human reviewer never sees them while the agent still reads them. Manual review of the hidden content is recommended.Fix: Remove the large whitespace padding (blank-line blocks or long space runs) and review any content hidden below or to the right of it. Keep skill files compact and reviewable so no instructions can be
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00220 | $0.04919 |
| Opus 5 | $0.00110 | $0.02459 |
| Sonnet 5 | $0.00044 | $0.00984 |
| Haiku 4.5 | $0.00022 | $0.00492 |
Grade A, and why
fixing-flaky-tests scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 8d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 298 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Fixing flaky tests
Before you propose a change to how the suite runs in CI, check things already tried. It records measured verdicts on test parallelism, sharding, and coverage-based selection, so a rejected approach is not rebuilt.
Three non-negotiables, in order:
- Reproduce before you fix. A fix for a failure you never observed is a guess. Only fall back to analytical fixes when the escalation ladder below is exhausted.
- Fix the root cause. Sleeps, raised timeouts, retries, and weakened assertions hide flakes; they do not fix them.
- Validate with an N-run loop. One green run proves nothing about an intermittent failure. Size N to the observed failure rate.
Stabilizing the test is not the only valid ending. Once you know why it flakes, step 5 asks whether it should exist. A test that catches nothing real is worth deleting, and one that flakes because of the level it runs at is worth moving down a rung.
Before any of these: measure, don't assume. Flaky-vs-deterministic, and the rate, are facts to establish from verifiable GitHub run data (step 1) — never inherited from a Slack alert, a teammate's guess, or a ci:insights label.
For triaging a red CI run (finding and classifying the failure), use the debugging-ci-failures skill first — this skill takes over once the failure is classified as a flaky test.
For writing new Playwright tests that aren't flaky, use the playwright-test skill.
investigating-ci-failures (green/red boundary) and diagnosing-ci-and-merge-bottlenecks (the engineering-analytics-flaky-tests tool's caveats) are product skills under products/engineering_analytics/skills/, not invocable here: read their SKILL.md at that path.
1. Measure the failure rate — from GitHub, not from a digest
The GitHub Actions API (or GitHub MCP) is the source of truth. hogli ci:insights is a digest, not an oracle — it can mislabel flaky-vs-deterministic, it lags the API until GitHub's webhook settles, and it cannot give you a rate at all: it reports absolute counts, because CI emits failures but omits ordinary passing runs, so there is no denominator. Use it to validate a hypothesis or pull historical context, never as the first move or the classification authority.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 8d ago First seen · 298 lines · 220 tokens per session scan A 7dfddbf3bbd0
fixing-flaky-tests is a skill published in the GitHub repository PostHog/posthog-foss (714 stars, last pushed today), licensed MIT. It adds 220 tokens to every session and 4,919 once invoked, about $0.0011 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
systematic-debugging
4-phase root cause debugging: understand bugs before fixing.
langsmith-observability
LLM observability platform for tracing, evaluation, and monitoring. Use when debugging LLM applications, evaluating model outputs against datasets, monitoring production systems, or building systematic testing pipelines for AI applications.
experimental-code-coverage-local-debugger
Runs code coverage locally via Universal Test Runner (UTR) or helper scripts, mimicking LUCI trybots. Activate when CQ tryjobs fail or underreport coverage, to test local GN/recipe repairs before uploading, or to debug hermetic crashes.
adversarial-reviewer
Adversarial code review that assumes bugs exist and hunts for them. Use when asked to review code, find bugs, audit for correctness, stress-test a PR, or when someone says "tear this apart" or "what's wrong with this". Give no benefit of the doubt — every line is guilty until proven innocent.
cli-e2e
Write, modify, or debug Docker-based Composio CLI end-to-end tests under ts/e2e-tests/cli, including binary invocation, fixture isolation, output assertions, and package manifests. Use for CLI E2E test suites only; use cli-command for CLI source implementation.
ios-simulator
Verify and debug native, React Native, Expo, or Flutter apps on an iOS Simulator with agent-device. Use when an agent needs to launch an app, inspect its live UI, tap, type, scroll, validate a code change, collect failure evidence, or reproduce a workflow on an iPhone or iPad Simulator.