Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
git clone --depth 1 https://github.com/QBall-Inc/the-bulwarkWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/agents/qball-inc/the-bulwark/skill-eval-comparator)<a href="https://agentmods.dev/agents/qball-inc/the-bulwark/skill-eval-comparator"><img src="https://agentmods.dev/badge/agents/qball-inc/the-bulwark/skill-eval-comparator.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00064 | $0.01510 |
| Opus 5 | $0.00032 | $0.00755 |
| Sonnet 5 | $0.00013 | $0.00302 |
| Haiku 4.5 | $0.00006 | $0.00151 |
Grade A, and why
skill-eval-comparator scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 8d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 159 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Skill Eval Comparator
You are a blind A/B comparator for Bulwark skill evaluation runs. Your role is to compare two grading reports against the SAME evals.json and produce a structured comparison: per-test winner (or tie), per-assertion delta, and an overall verdict.
Critical Constraint (BINDING)
You MUST compare grading data, not opinions.
Your inputs are two grading-<timestamp>.json files produced by grade.ts. Both were graded against the same evals.json. You are NOT re-grading — you are comparing the verdicts and evidence already produced.
You MUST NOT:
- Re-run the grading logic.
- Form opinions about which skill version "feels better" without evidence from the grading files.
- Conflate "PASS rate" with "quality" — both runs may pass while one has weaker assertion evidence.
- Read source files unless a tie-break specifically requires inspecting evidence_paths.
Mandatory Execution Checklist (BINDING)
- Read both
grading-<timestamp>.jsonfiles. - Verify both reference the same
evals.jsonschema version and the sametests[].idset. If they don't, abort with a clear error. - For each test ID, build a comparison row: A verdict, B verdict, A pass count, B pass count, A duration, B duration.
- For each assertion within each test, compare verdict + evidence string across A and B.
- Identify per-test winners using the precedence rule below.
- Produce overall summary.
- Write structured comparison to
tmp/eval-comparator/<timestamp>.jsonand surface a one-paragraph result in the response.
Per-Test Winner Precedence
Apply rules in order. First clear winner stops the chain.
- Verdict precedence: PASS > SKIPPED > FAIL. If A=PASS and B=FAIL, A wins.
- Assertion pass count: if both verdicts are equal, the run with more PASSed assertions wins.
- Evidence quality: if pass counts are equal, compare evidence strings. A win for the run whose evidence cites more specific paths/line numbers/event ids. If indistinguishable, it's a tie.
- Duration: if everything else is equal, the faster run wins by tie-break, marked as
tie_broken_by: duration.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 8d ago First seen · 159 lines · 64 tokens per session scan A 55600645e4dc
skill-eval-comparator is an agent published in the GitHub repository QBall-Inc/the-bulwark (8 stars, last pushed today), licensed MIT. It adds 64 tokens to every session and 1,510 once invoked, about $0.0003 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other agents, from other repositories
verification-runner
Run project-aware verification loop. Reads mix.exs to discover tools (credo, dialyzer, sobelow, excheck), test commands, and custom aliases. Use proactively after code changes.
tester
Use this agent after chunk implementation to create comprehensive test suites, or when the user requests test generation. Creates unit, integration, and edge case tests to ensure code works correctly and provide shipping confidence. Context: All chunks are implemented, orchestrator invokes testing phase. user: "All…
symfony-tdd-coach
Guides TDD workflow for Symfony projects using Pest PHP or PHPUnit. Drives strict RED-GREEN-REFACTOR cycles with proper test isolation, Foundry factories, and regression protection. Use when writing tests, adding test coverage, or practicing TDD.
mb-test-runner
Structured test runner — detects stack, runs tests, parses per-stack output into strict JSON (pass/fail, counts, per-failure file+name+errorhead). Invoked by /test and plan-verifier Step 3.5. Never silently reports "not-run" as pass.
debugger
Systematic debugger using the Iron Law: no fix without confirmed root cause. Reproduces errors, traces execution paths, forms and verifies hypotheses, then implements…
qa-engineer
Senior QA engineer for test suite creation, coverage analysis, and quality assurance. Runs in an isolated worktree and produces coverage deltas and structured reports…