Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add ericrisco/rsc-harness --skill agent-evalgit clone --depth 1 https://github.com/ericrisco/rsc-harnessWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/ericrisco/rsc-harness/agent-eval)<a href="https://agentmods.dev/skills/ericrisco/rsc-harness/agent-eval"><img src="https://agentmods.dev/badge/skills/ericrisco/rsc-harness/agent-eval/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/ericrisco/rsc-harness/agent-eval"><img src="https://agentmods.dev/badge/skills/ericrisco/rsc-harness/agent-eval.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- NVIDIA SkillSpector pass
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00079 | $0.03109 |
| Opus 5.5 | $0.00032 | $0.01244 |
| Sonnet 5.5 | $0.00016 | $0.00622 |
| Haiku 4.5 | $0.00008 | $0.00311 |
Grade A, and why
agent-eval scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 188 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Measure agent quality you can defend and gate on
Turn "the agent feels better" into a number you can put in a PR check. You own the eval dataset, the scorer mix, the LLM-as-judge calibration, and the block-on-regression CI gate — framework-neutral, provider-neutral.
Do NOT use — route instead
| The ask | Route to | Why it is not this skill |
|---|---|---|
| Build the agent loop, tools, RAG plumbing | building-agents |
It builds the system; you score it. They cross-link. |
| "Make the answers shorter / rewrite the prompt" | prompt-engineering |
Evals say it is worse; that skill changes the words. You never edit the prompt. |
| pytest/jest on deterministic functions | testing-py / testing-web |
Assert-equals on pure code, not stochastic outputs scored by a judge. |
| Dashboards / tracing of live production traffic | observability |
Online monitoring; you are offline + pre-merge. |
| Red-team, jailbreak, prompt injection | agent-safety |
Adversarial coverage, not quality measurement. |
| Per-token cost budgets and accounting | cost-tracking |
You report cost-per-task as one metric; the discipline lives there. |
| A/B stats on product/funnel metrics | ab-testing |
Web experiments, not offline model comparison on a fixed set. |
The eval anatomy
Every framework instantiates the same five-stage pipeline. Learn it once; the tool is a detail.
dataset ──▶ runner ──▶ scorers ──▶ metrics ──▶ gate
(JSONL (calls the (det / judge (aggregate + (pass/fail
golden system per / human) bootstrap CI) exit code)
set) case)
DeepEval, Inspect AI, and promptfoo are all just opinionated wrappers around this. If you understand the stages you can switch tools without relearning the craft.
Build the dataset first
Build your own golden set — a public leaderboard number is not your number, because identical model weights swing SWE-Bench Verified by 10–20 points just by changing the harness. Measure your task on your data. The dataset is the asset; everything else is replaceable. Rules:
What ships with it
5 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 1mo ago First seen · 188 lines · 79 tokens per session scan A 769f75907b41
agent-eval is a skill published in the GitHub repository ericrisco/rsc-harness (231 stars, last pushed today), licensed MIT. It adds 79 tokens to every session and 3,109 once invoked, about $0.0003 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
browser-qa
Testing web apps through a real browser: reconnaissance before acting, selector discovery, state capture, console errors and screenshot evidence. Use on a web surface end to end, or to reproduce a front-end bug.
test-driven-development
Writing code with tests first: RED-GREEN-REFACTOR, reproduce bugs as failing tests before fixing, and avoid testing anti-patterns. Use when implementing behaviour, fixing a defect, or asked to add tests.
verification-before-completion
Proving work works before claiming it does: exercise the real path, read the actual output, gather fresh evidence per claim, and say what remains unverified. Use before reporting done, handing off, or posting results.
llm-evaluator
Evaluate LLM outputs systematically using LLM-as-judge, human evaluation frameworks, and regression testing. Use when assessing model quality, comparing models, or preventing quality regression.
evaluating-llms-harness
Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.
llm-as-judge-evaluation
Evaluate LLM outputs using frontier models as judges. Use for pairwise model comparison, quality scoring with custom rubrics, and automated evaluation pipelines. Covers position bias mitigation, statistical significance, and generating preference data for DPO/RLHF.