Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/ssheleg/agent-stack/agent-evalsnpx skills add ssheleg/agent-stack --skill agent-evalsgit clone --depth 1 https://github.com/ssheleg/agent-stackWhat it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00219 | $0.03787 |
| Opus 5 | $0.00110 | $0.01894 |
| Sonnet 5 | $0.00044 | $0.00757 |
| Haiku 4.5 | $0.00022 | $0.00379 |
Grade A, and why
agent-evals scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 316 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Agent evals — proving the thing behaves
An agent's behaviour does not exist in its source. The code says what it is allowed to do; only a run says what it did. So the artifact under test is the execution record, and the suite runs on two clocks: the observable is authored up front, before the implementation exists, and the corpus it runs against is grown from production (§6).
Three claims follow, and they are what makes this different from testing ordinary code:
- You are testing reasoning, not code paths, so one granularity is never enough.
- Every natural-language input is unique, so the edge cases cannot be enumerated offline. Production is not only where you catch what you missed — it is where you discover what to test for.
- Traces become test cases. The corpus grows from what actually happened; the criterion it is measured against does not.
1. Three primitives
| Primitive | Is | Carries |
|---|---|---|
| Run | one model call | the complete prompt — instructions, tools, context — and what came back |
| Trace | one full execution | every run, every tool call with arguments and results, nested to show how steps relate |
| Thread | many traces in one session | multi-turn context in order, state evolution (memory, files, artifacts), and elapsed time |
The thread level is the one most systems lack, and it is where a whole class of defect lives: a bad memory write at turn 6 surfaces as a wrong answer at turn 11, and neither the run nor the trace view can localise it.
Precondition for all of this: traces are durable and queryable. A live event stream renders a progress bar and evaporates. If you cannot fetch last week's low-scoring runs by id, nothing below is available to you — see §7.
2. Three granularities
Each primitive gets its own kind of assertion.
Single-step — validates a run
Fixture is a serialized run: prompt, tool schemas, context. Assert the decision at that point — tool name, argument shape.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 2d ago First seen · 316 lines · 219 tokens per session scan A b62e257fe396
agent-evals is a skill published in the GitHub repository ssheleg/agent-stack (2 stars, last pushed 2d ago), licensed MIT. It adds 219 tokens to every session and 3,787 once invoked, about $0.0011 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
cotal-setup
Set up Cotal on this machine: install it, start a local agent mesh (NATS + JetStream), verify it, and put an agent on it. Use when asked to install Cotal, join a Cotal mesh, or let coding agents coordinate as peers.
team-topology
Define a multi-agent team for ANY task on ANY system as an explicit deployment topology - pick the shape from the task's dominant risk, specify the runtime/communication/trust layers, place model capability by lane, present it as a diagram + table + trust-boundary note + open choices, and deploy ONLY after the user…
cli-plugins-skills
Manage Omni Skills (list, install, test, remove), plugins (create, configure), and persistent memory (search, add, clear) from the CLI.
cli-a2a
Interact with the OmniRoute A2A server from the CLI. Send tasks, inspect skill execution history, and test the JSON-RPC 2.0 agent-to-agent protocol interactively.
mcp-builder
Design, build, configure, or debug Model Context Protocol servers for codewhale, including stdio and HTTP/SSE transports.
omni-github-skills
Search, score, scan, and import agent skills from GitHub repositories that contain SKILL.md, CLAUDE.md, .cursorrules, and similar agent skill files. Discover community skills across many tool and provider categories, evaluate relevance with heuristic scoring, check for malware or hardcoded secrets, and install into…