Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add ssheleg/agent-stack --skill agent-evalsgit clone --depth 1 https://github.com/ssheleg/agent-stackWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/ssheleg/agent-stack/agent-evals)<a href="https://agentmods.dev/skills/ssheleg/agent-stack/agent-evals"><img src="https://agentmods.dev/badge/skills/ssheleg/agent-stack/agent-evals/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/ssheleg/agent-stack/agent-evals"><img src="https://agentmods.dev/badge/skills/ssheleg/agent-stack/agent-evals.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00076 | $0.04813 |
| Opus 5.5 | $0.00030 | $0.01925 |
| Sonnet 5.5 | $0.00015 | $0.00963 |
| Haiku 4.5 | $0.00008 | $0.00481 |
Grade A, and why
agent-evals scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 376 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Agent evals — proving the thing behaves
An agent's behaviour does not exist in its source. The code says what it is allowed to do; only a run says what it did. So the artifact under test is the execution record, and the suite runs on two clocks: the observable is authored up front, before the implementation exists, and the corpus it runs against is grown from production (§6).
Three claims follow, and they are what makes this different from testing ordinary code:
- You are testing reasoning, not code paths, so one granularity is never enough.
- Every natural-language input is unique, so the edge cases cannot be enumerated offline. Production is not only where you catch what you missed — it is where you discover what to test for.
- Traces become test cases. The corpus grows from what actually happened; the criterion it is measured against does not.
Triggers
"trajectory eval", "checker node", "is the agent getting better", "оценка агента", "как проверить агента".
Moved here from the description in 0.31.3: "регрессионный набор".
1. Three primitives
| Primitive | Is | Carries |
|---|---|---|
| Run | one model call | the complete prompt — instructions, tools, context — and what came back |
| Trace | one full execution | every run, every tool call with arguments and results, nested to show how steps relate |
| Thread | many traces in one session | multi-turn context in order, state evolution (memory, files, artifacts), and elapsed time |
The thread level is the one most systems lack, and it is where a whole class of defect lives: a bad memory write at turn 6 surfaces as a wrong answer at turn 11, and neither the run nor the trace view can localise it.
Precondition for all of this: traces are durable and queryable. A live event stream renders a progress bar and evaporates. If you cannot fetch last week's low-scoring runs by id, nothing below is available to you — see §7.
2. Three granularities
What ships with it
4 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 2d ago Changed · +1 lines · -31 tokens per session 8238e5d9e4b8
- 4d ago Changed · +1 lines 6c17dff16c45
- 8d ago Changed · +9 lines · -106 tokens per session 221d4487a177
- 15d ago Changed · -6 tokens per session 8f533ce3e320
- 18d ago Changed · +1 lines e565a39ab596
- 1mo ago Changed · +17 lines e69a6021140f
- 1mo ago Changed · +31 lines 18c38888b665
- 1mo ago First seen · 316 lines · 219 tokens per session scan A b62e257fe396
agent-evals is a skill published in the GitHub repository ssheleg/agent-stack (4 stars, last pushed yesterday), licensed MIT. It adds 76 tokens to every session and 4,813 once invoked, about $0.0003 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
install-verification
Use when verifying that a generated agent package can be installed, discovered by runtimes, and checked without private dependencies.
benchmark-critic
Evaluate benchmark design for fair baselines, measurable outcomes, leakage, cherry-picking, and reproducibility.
release-readiness
Check whether a change is ready for a public release candidate across packaging, docs, tests, and rollback risk.
agent-puzzles
Competitive puzzle arena for AI agents with timed solving, per-model leaderboards, and 5 categories (reverse captcha, geolocation, logic, science, code). Use when solving puzzles, tracking rankings, creating new challenges, or benchmarking agent capabilities.
agent-evaluation-v2
Agent Evaluation workflow skill. Use this skill when the user needs Testing and benchmarking LLM agents including behavioral testing, and the operator should preserve the upstream workflow, copied support files, and provenance before merging or handing off.
agent-evaluation
Agent Evaluation workflow skill. Use this skill when the user needs Testing and benchmarking LLM agents including behavioral testing, and the operator should preserve the upstream workflow, copied support files, and provenance before merging or handing off.