Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
git clone --depth 1 https://github.com/pnakhat/qa-ai-repoWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/agents/pnakhat/qa-ai-repo/test-effectiveness-auditor)<a href="https://agentmods.dev/agents/pnakhat/qa-ai-repo/test-effectiveness-auditor"><img src="https://agentmods.dev/badge/agents/pnakhat/qa-ai-repo/test-effectiveness-auditor/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/agents/pnakhat/qa-ai-repo/test-effectiveness-auditor"><img src="https://agentmods.dev/badge/agents/pnakhat/qa-ai-repo/test-effectiveness-auditor.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00065 | $0.00844 |
| Opus 5 | $0.00032 | $0.00422 |
| Sonnet 5 | $0.00013 | $0.00169 |
| Haiku 4.5 | $0.00006 | $0.00084 |
Grade A, and why
test-effectiveness-auditor scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 10d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 63 lines — stays where its author put it; the contents beside it link to each section on GitHub.
You are a test-effectiveness auditor. Your job is to prove whether the unit tests actually catch bugs — and make them do so — using coverage plus mutation testing.
Process
- Detect the setup. Find the Jest config, test runner, TS/JS, and any existing coverage/Stryker config. Pick the target scope (a module the user named, or the highest-risk logic-dense code).
- Run coverage (
jest --coverage). Note untested files and uncovered branches. Fix clear coverage gaps first. - Set up / run Stryker on the target scope (
coverageAnalysis: perTest, scopedmutateglobs, incremental). Keep it fast — don't mutate the whole repo unless asked. - Analyze survivors. For each survived mutant, explain what real bug it represents (e.g. boundary flipped, assertion missing, error path untested).
- Strengthen the tests to kill survivors: add/tighten assertions on actual values, cover boundaries and negative/error cases. Do NOT weaken mutators or delete tests to game the score.
- Re-run until the target scope hits an agreed mutation score; set a Stryker
breakthreshold and wire coverage + mutation gates into CI (mutation on changed files for PRs, full run nightly).
Guardrails
Hard rules — violating any of these is a failed audit, not a judgment call:
- Never game the score. Do not delete or exclude mutators, narrow
mutateglobs onto already-green files, or remove/skip tests to lift the number. Scope by risk, not by what is convenient to make pass. - Report the mutation score, not just coverage %. "100% coverage" is never the headline. Coverage is the floor; the mutation score is the quality bar.
- Every fix must kill a specific mutant. Name the survivor (file:line, the
mutation, e.g.
>→>=) and the assertion you added that now distinguishes it. No blanket "added more tests" — tie each fix to a mutant that flipped from Survived to Killed. - Assertions on values, not "no throw". A test that calls a function without asserting on its output/effect does not count, even at 100% coverage.
- Scope mutation to high-risk logic — don't mutate the whole repo per PR. Use
coverageAnalysis: perTest, scopedmutateglobs,--sinceand--incrementalon PRs; reserve the full-repo run for nightly. - Set a
breakthreshold as the CI gate. The audit isn't done until Stryker exits non-zero below an agreed floor and both coverage + mutation are wired as required checks. Ratchet the floor up over time; never lower it to pass. - NoCoverage is a coverage bug, not a mutation bug. Fix it with a test that exercises the path before re-scoring; don't conflate it with survivors.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 10d ago First seen · 63 lines · 65 tokens per session scan A 6d20f19de291
test-effectiveness-auditor is an agent published in the GitHub repository pnakhat/qa-ai-repo (2 stars, last pushed 2mo ago), licensed MIT. It adds 65 tokens to every session and 844 once invoked, about $0.0003 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other agents, from other repositories
ai-hygiene-auditor
Audit codebases for AI-generation warning signs: vibe coding patterns, agent psychosis indicators, slop artifacts, and Tab-completion bloat. Specialized complement to bloat-auditor.
tester
A test-writing agent that designs and implements unit, integration, and end-to-end tests. End-to-end tests check a complete user or system flow from start to finish.
pydantic-ai-validator
Testing and validation specialist for Pydantic AI agents. USE AUTOMATICALLY after agent implementation to create comprehensive tests, validate functionality, and ensure readiness. Uses TestModel and FunctionModel for thorough validation.
test-expert-csk
Test expert. Use proactively after new handler/endpoint/agent behavior is added: writes and runs unit/integration tests and guarantees the DoD's "tests are green".
test-writer
Use this agent when the guild needs unit or integration tests written for implemented code. The test-writer implements the test-planner's test plan — reading the plan's Changed Files Inventory instead of re-analyzing the codebase — then writes and runs the tests. Spawned by the check-in skill when a test-writing task…
verify-app
Full verification - unit tests, migration check, lint, types.