Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add bikeread/promethos --skill build-agent-evalsgit clone --depth 1 https://github.com/bikeread/promethosWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/bikeread/promethos/build-agent-evals)<a href="https://agentmods.dev/skills/bikeread/promethos/build-agent-evals"><img src="https://agentmods.dev/badge/skills/bikeread/promethos/build-agent-evals/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/bikeread/promethos/build-agent-evals"><img src="https://agentmods.dev/badge/skills/bikeread/promethos/build-agent-evals.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00024 | $0.00778 |
| Opus 5 | $0.00012 | $0.00389 |
| Sonnet 5 | $0.00005 | $0.00156 |
| Haiku 4.5 | $0.00002 | $0.00078 |
Grade A, and why
build-agent-evals scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 9d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 88 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Goal
Create a small but meaningful evaluation set that can catch regressions and false confidence early.
Inputs
- Agent goals and key behaviors
- Known failure modes
- Available test harnesses, traces, or manual checks
Non-Goals
- Building a giant benchmark suite before the core workflow is stable
- Equating one happy-path demo with real evaluation coverage
Workflow
Trigger signals
- Agent is nearing completion but has no test cases
- User asks "怎么验" or "how do we know it works"
- User asks for the "smallest eval pack", "minimal eval harness", or "what should we test first"
- Behavior was recently changed and nothing checks for regressions
- Agent has been deployed but no one defined what "working correctly" means
1. Choose the behaviors that matter most
Identify the actions or decisions that would make the agent untrustworthy if they failed: planning quality, tool use, recovery, memory handling, approval respect, or output correctness. Rank them by user impact and by how likely they are to regress. Success criteria: The eval scope is centered on real risk, not on what is easy to test.
2. Build a minimal scenario set
Create a compact mix of cases that covers the important behaviors: a normal success, an ambiguous request, a tool failure, a recovery path, and any context-pressure case that is part of the real task. If the agent can take consequential actions, include at least one negative case for refusal, checkpointing, escalation, or approval-boundary compliance. If the user asks for the smallest useful pack, cut optional cases aggressively instead of letting the battery drift into a benchmark wishlist. Keep each scenario specific enough that a human can tell what good behavior looks like. Success criteria: The harness exercises the agent across success, failure, and recovery instead of only the easiest path.
3. Define scoring and judgment rules
Write what counts as pass, concerns, fail, or unacceptable, and note what artifacts a reviewer should inspect to make the call consistently. If a metric is used, explain what behavior it is actually standing in for. Success criteria: Another reviewer can score the same run without inventing their own rubric.
What ships with it
1 file beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 9d ago First seen · 88 lines · 24 tokens per session scan A 0fd14c9dc7df
build-agent-evals is a skill published in the GitHub repository bikeread/promethos (33 stars, last pushed 5mo ago), licensed MIT. It adds 24 tokens to every session and 778 once invoked, about $0.0001 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
research-engineer
An uncompromising Academic Research Engineer. Operates with absolute scientific rigor, objective criticism, and zero flair. Focuses on theoretical correctness, formal verification, and optimal implementation across any required technology.
tika-eval-compare
Compare extracts from two Tika builds over a corpus to detect regressions in content, encoding, exceptions, and embedded-document handling. Use for "compare before/after extracts", "eval this change against the corpus".
neuron-evaluation-engineer
Create and run AI evaluations with datasets, assertions, and output drivers in Neuron AI. Use this skill whenever the user mentions evaluation, testing AI systems, creating evaluators, dataset-driven testing, assertion-based validation, or wants to measure AI system performance. Also trigger for tasks involving…
jetson-validate-image
Use after jetson-flash-image to run static BSP checks, on-target smoke/regression tests on a flashed DUT, or both. Not for build or flash steps. Triggers: validate bsp, on-target validation.
atmos-validation
Validate Atmos projects, components, arbitrary JSON Schema inputs, EditorConfig, and GitHub Actions; use affected-file selection and native CI annotations.
skill-benchmark
Benchmark AI skill effectiveness by measuring implementation quality against legacy constraints.