Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/ericrisco/rsc-harness/agent-evalnpx skills add ericrisco/rsc-harness --skill agent-evalgit clone --depth 1 https://github.com/ericrisco/rsc-harnessWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/ericrisco/rsc-harness/agent-eval)<a href="https://agentmods.dev/skills/ericrisco/rsc-harness/agent-eval"><img src="https://agentmods.dev/badge/skills/ericrisco/rsc-harness/agent-eval.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00079 | $0.03109 |
| Opus 5 | $0.00039 | $0.01554 |
| Sonnet 5 | $0.00016 | $0.00622 |
| Haiku 4.5 | $0.00008 | $0.00311 |
Grade A, and why
agent-eval scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 5d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 188 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Measure agent quality you can defend and gate on
Turn "the agent feels better" into a number you can put in a PR check. You own the eval dataset, the scorer mix, the LLM-as-judge calibration, and the block-on-regression CI gate — framework-neutral, provider-neutral.
Do NOT use — route instead
| The ask | Route to | Why it is not this skill |
|---|---|---|
| Build the agent loop, tools, RAG plumbing | building-agents |
It builds the system; you score it. They cross-link. |
| "Make the answers shorter / rewrite the prompt" | prompt-engineering |
Evals say it is worse; that skill changes the words. You never edit the prompt. |
| pytest/jest on deterministic functions | testing-py / testing-web |
Assert-equals on pure code, not stochastic outputs scored by a judge. |
| Dashboards / tracing of live production traffic | observability |
Online monitoring; you are offline + pre-merge. |
| Red-team, jailbreak, prompt injection | agent-safety |
Adversarial coverage, not quality measurement. |
| Per-token cost budgets and accounting | cost-tracking |
You report cost-per-task as one metric; the discipline lives there. |
| A/B stats on product/funnel metrics | ab-testing |
Web experiments, not offline model comparison on a fixed set. |
The eval anatomy
Every framework instantiates the same five-stage pipeline. Learn it once; the tool is a detail.
dataset ──▶ runner ──▶ scorers ──▶ metrics ──▶ gate
(JSONL (calls the (det / judge (aggregate + (pass/fail
golden system per / human) bootstrap CI) exit code)
set) case)
DeepEval, Inspect AI, and promptfoo are all just opinionated wrappers around this. If you understand the stages you can switch tools without relearning the craft.
Build the dataset first
Build your own golden set — a public leaderboard number is not your number, because identical model weights swing SWE-Bench Verified by 10–20 points just by changing the harness. Measure your task on your data. The dataset is the asset; everything else is replaceable. Rules:
What ships with it
5 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 5d ago First seen · 188 lines · 79 tokens per session scan A 769f75907b41
agent-eval is a skill published in the GitHub repository ericrisco/rsc-harness (60 stars, last pushed 2d ago), licensed MIT. It adds 79 tokens to every session and 3,109 once invoked, about $0.0004 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
research-tool-updates
Research recent upstream releases of every rulesync target tool, detect capabilities rulesync has not yet followed, file one GitHub issue per tool for the gaps, and scout popular or promising coding agents rulesync does not target yet.
review-pr
Walk Sean through an incoming waku-agent PR or issue and present it his way — four fixed sections: what this is, why it matters, how HE can test it with you as copilot, and are we ready to merge / reply / close and why. Use whenever Sean asks to look at, test, triage, or decide on a pull request or an issue, and…
batch-all-issues
Resolve every open issue one at a time: fact-check each with web research, close the ones that need no action, and run the goal-pr skill to fix, review, and merge the ones that do — repeating until no actionable issues remain.
excalidraw
Generate an Excalidraw whiteboard in Sean's hand-drawn video style (Excalifont, roughness 1, green signature, socials + watermark, source labels). Use whenever the user wants a whiteboard, diagram, teaching board, or "chart" for a video or the docs/whiteboards gallery — anything Sean will film with.
create-issue-with-websearch
Create a GitHub issue from a vague idea by thoroughly researching the topic on the web to sharpen the specification before filing.
draft-issue-to-external-repository
Draft a GitHub issue targeting a third-party repository (outside this project's owner). Researches the target repo's conventions and the topic in depth, iterates with the user on scope and tone, and produces a final body saved to tmp/external-repository-issues/.md for the user to post manually when gh auth lacks…