analyze_flaky_tests

A workflow for investigating automated tests that sometimes pass and sometimes fail, a problem known as flaky testing. It can either report likely causes or apply fixes after the user approves them.

In plain words
What is it for?
Use it to analyze flaky automation tests, propose stabilization changes, or repair locators and verify the result in its fix mode.
Why use it?
It replaces guesswork with analysis of test logs, code, and the actual page structure, helping identify why results are inconsistent.

Command for Claude Code

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

agentmods
npx agentmods add commands/anhtester/claude-testing-kit/analyze_flaky_tests
Clone the repo
git clone --depth 1 https://github.com/anhtester/claude-testing-kit

Made for: Claude Code.

Per session 52 Only the description is in the session, so the agent can decide to use it. The body loads when it is invoked.
When invoked 3,128 The whole file, excluding the scripts and references it only reads on demand.
Security scan A 0 findings. Scan, not verified.
Origin original No closer match found in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5 $0.00052 $0.03128
Opus 5 $0.00026 $0.01564
Sonnet 5 $0.00010 $0.00626
Haiku 4.5 $0.00005 $0.00313

Measured 2d ago against content hash 67f47d7a9052, method: parsed. Prices are Anthropic first-party input rates as of 2026-08-30, from the pricing page.

Security

Grade A, and why

analyze_flaky_tests scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

Origin

Copies of this mod

1 near-identical copy found in the catalogue:

.claude/commands/analyze_flaky_tests.md · 244 lines

How it starts

The opening of the file, as written. The whole thing — 244 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Workflow: Phân Tích & Khắc Phục Flaky Tests

BẮT BUỘC (MANDATORY SKILL): Bạn PHẢI nạp và đọc kỹ nội dung của skill skills-flaky-test-analyzer (tại .claude/skills/skills-flaky-test-analyzer/SKILL.md) trước khi bắt đầu. Ngoài ra, tham khảo thêm skill skills-smart-locator-agentskills-locator-healer-agent khi cần sửa/thay locator.

Workflow này giúp agent tự động phân tích các automation test không ổn định (lúc pass lúc fail), xác định root cause chính xác, và (tùy mode) tự động sửa code để stabilize test.

⚠️ Nguyên tắc thực thi

  • Tất cả output bằng Tiếng Việt
  • KHÔNG đoán nguyên nhân — phải phân tích log, code, và DOM thực tế
  • Phải chờ user xác nhận danh sách fix tại Bước 3 trước khi sửa code (Mode FIX)
  • Nếu user chưa cung cấp test file/error log → hỏi trước khi bắt đầu
  • ⚠️ Sau khi user xác nhận fix → agent tự sửa + verify, KHÔNG hỏi lại trong quá trình fix

2 Chế độ (Mode)

Mode Khi nào sử dụng Output
ANALYZE (mặc định) User cần hiểu nguyên nhân flaky, chưa muốn sửa code Báo cáo phân tích + Đề xuất fix
FIX User muốn agent tự sửa code luôn Như ANALYZE + Code đã sửa + Kết quả verify

Nếu user nói "sửa luôn", "fix giùm", "khắc phục test này", hoặc yêu cầu agent sửa code → tự động chuyển sang Mode FIX.

Input cần thu thập

Agent cần ít nhất 1 trong các input sau từ user:

Input Cách lấy Độ ưu tiên
Test file path User cung cấp hoặc agent tìm trong project ⭐ Bắt buộc
Error log / stack trace User paste hoặc agent chạy test để thu thập ⭐ Bắt buộc
CI/CD log User cung cấp URL hoặc file log Tùy chọn
Test report (HTML/JSON) Playwright report, Allure, TestNG report Tùy chọn
Lịch sử fail Số lần fail / tổng số lần chạy Tùy chọn

Các bước thực hiện

Bước 1: Thu thập thông tin & Tái hiện lỗi (Detect & Reproduce)

  1. Đọc test file bằng Read:
    • Xác định framework (Playwright / Selenium / Appium / Pytest / TestNG)
    • Ghi nhận cấu trúc: Page Objects, fixtures, helper functions
    • Đánh dấu các vùng code nghi ngờ (waits, locators, assertions, setup/teardown)

Read the full file on GitHub · 244 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 2d ago First seen · 244 lines · 52 tokens per session scan A 67f47d7a9052

Subscribe to this mod's changes

analyze_flaky_tests is a command published in the GitHub repository anhtester/claude-testing-kit (49 stars, last pushed 24d ago), licensed MIT. It adds 52 tokens to every session and 3,128 once invoked, about $0.0003 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.