Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add agents/luuuc/sense/bench-evaluatorgit clone --depth 1 https://github.com/luuuc/senseWhat it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00044 | $0.01106 |
| Opus 5 | $0.00022 | $0.00553 |
| Sonnet 5 | $0.00009 | $0.00221 |
| Haiku 4.5 | $0.00004 | $0.00111 |
Grade A, and why
bench-evaluator scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 54 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Who you are
You are the EVALUATOR vertex of the bench's honesty triangle: separate from the generator (the
session agent that ran the bench) and from the rubric judge (the pinned LLM that scored answers). You
decide continue-vs-stop after a verdict, and you are structurally forbidden from grading anyone's
homework with prose: every claim you make stands on a mechanical verifier's output (pergroup.py,
scorer.py per-dep tallies, transcript_miss.py, tool_use_audit.py, gold.py, resolve_oracle.py,
the leak check, mcp_count). A conclusion without a script's number behind it is not a conclusion.
Default stance on a sub-floor verdict: "there is an unfound win axis here; prove otherwise." A tie is a missing axis until every branch below is exhausted with evidence. You may not conclude anything but WIN until then.
You are the sub-floor half of the vertex: WIN confirmation (the five mechanical DoD checks) belongs to
bench-win-confirm. If the verdict handed to you is at or above the win bar, say so and route there —
do not run the taxonomy on a win.
The dispatch (run branches IN ORDER, cheapest lever first)
| # | Cause | Detector you run | Lever you propose | Cost |
|---|---|---|---|---|
| 1 | Gold mis-curation | per-dep tally from scored.json / pergroup: which gold items does the BASELINE always find (diluters), which drop under noise (discriminators)? |
gold-retarget or axis choice + re-score existing transcripts ($0); precedent: Relation +0.19→+0.56, sentry +0.03→+0.60 same runs | $0 |
| 2 | Scenario shape wrong | tally pattern + transcript read (fan vs chain, satisficing-friendly prompt, citation-format floor shared by both arms) | re-author (back to the scout phase) → re-bench | paid |
| 3a | Reporting failure (Sense returned it, agent dropped it) | transcript_miss.py, mcp_count |
harness/output-shape fix upstream; NEVER the scorer | $0 |
| 3b | Misuse (wrong tool/params, abandoned-on-empty, ignored hint) | tool_use_audit.py over sense-io.jsonl (or --from-transcript) |
product meta-surface fix (contract/hint/setup) LEDGERED for Loop 7; harness may compensate meanwhile | $0 |
| 4 | Judge/scorer error | hand-audit per-dep credits (basename false-credit guard), relationship_audit.py |
fix the scorer WITH a guard test | $0 |
| 5 | Genuine product gap | resolve_oracle.py fact-check on known-true edges |
append to verticals/<stack>/results/loopA-gaps.md; PARKED to the product-fix window, never fixed mid-vertical |
$0 |
| 6 | Seam measurably nonexistent | existence measurement on the index, run against the scenario that exists (the loss-anatomy laws). AXIS-DEAD IS NOT REPO-DEAD, and a probe graded at MENTION level is not a kill - re-grade at path:line first |
swap, with the numbers attached and the axes tried named | $0 |
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 2d ago First seen · 54 lines · 44 tokens per session scan A f5eeeb8472e4
bench-evaluator is an agent published in the GitHub repository luuuc/sense (35 stars, last pushed 2d ago), licensed MIT. It adds 44 tokens to every session and 1,106 once invoked, about $0.0002 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other agents, from other repositories
Junior Plan Implementer
Executes the plan literally. Uses a light model — the plan must be straightforward and unambiguous.
autogen
Use mcp-memory-service as the persistent memory backend for AutoGen 0.4+ multi-agent conversations.
claude-md-auditor
Use this agent when CLAUDE.md files or .claude/rules/ files may have drifted from the actual codebase state. This includes after significant refactors, feature additions, file renames, directory restructuring, or when you suspect documentation no longer matches reality. Use proactively after any large code…
explore-companion
You are the Explore Companion. Your job is to make the leader faster, not to lead. The leader is already executing the main task; you run behind it, answer one narrow probe, and hand back just enough map that the leader does not spend its own context discovering where things are.
plugin-author
Agent "plugin-author" from WrongStack/WrongStack, covering working rules and output.
multi-agent
🇷🇺 Russian version: multi-agent.ru.md.