Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add BuilderCed/agent-skills --skill agent-eval-frameworkgit clone --depth 1 https://github.com/BuilderCed/agent-skillsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/builderced/agent-skills/agent-eval-framework)<a href="https://agentmods.dev/skills/builderced/agent-skills/agent-eval-framework"><img src="https://agentmods.dev/badge/skills/builderced/agent-skills/agent-eval-framework/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/builderced/agent-skills/agent-eval-framework"><img src="https://agentmods.dev/badge/skills/builderced/agent-skills/agent-eval-framework.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00026 | $0.01163 |
| Opus 5.5 | $0.00010 | $0.00465 |
| Sonnet 5.5 | $0.00005 | $0.00233 |
| Haiku 4.5 | $0.00003 | $0.00116 |
Grade A, and why
agent-eval-framework scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 158 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Agent Evaluation Framework
When to Use
- Before deploying an agent to production
- After changing an agent's system prompt or skills
- When agent output quality seems to degrade
- During periodic quality reviews
- When comparing two agent configurations
Step 1: Define Evaluation Criteria
Choose criteria relevant to your agent's purpose:
Universal Criteria
| Criterion | Question | Score |
|---|---|---|
| Correctness | Is the output factually/technically correct? | 0-10 |
| Completeness | Does it cover all required aspects? | 0-10 |
| Relevance | Is every part relevant to the request? | 0-10 |
| Safety | Does it avoid harmful/insecure patterns? | 0-10 |
Code-Specific Criteria
| Criterion | Question | Score |
|---|---|---|
| Functionality | Does the code work as intended? | 0-10 |
| Edge Cases | Are edge cases handled? | 0-10 |
| Style | Does it match project conventions? | 0-10 |
| Security | Are there vulnerabilities? | 0-10 |
Content-Specific Criteria
| Criterion | Question | Score |
|---|---|---|
| Accuracy | Are claims supported by evidence? | 0-10 |
| Tone | Does it match the intended audience? | 0-10 |
| Structure | Is it well-organized? | 0-10 |
| Originality | Does it avoid generic/cliche content? | 0-10 |
Step 2: Choose Evaluation Method
A. Assertion-Based (Automated)
Define pass/fail conditions:
ASSERT: output contains "disclaimer"
ASSERT: output does NOT contain "TODO"
ASSERT: code compiles without errors
ASSERT: response length < 2000 tokens
ASSERT: no PII detected in output
Best for: Regression testing, CI/CD pipelines.
B. Reference-Based (Semi-Automated)
Compare output against a known-good reference:
- Exact match (strict)
- Semantic similarity (using embeddings)
- Key-point coverage (checklist)
Best for: Consistent tasks with known expected outputs.
C. Rubric-Based (Human + AI)
Score each criterion 0-10 with justification:
Correctness: 8/10 — Accurate but missed one edge case
Completeness: 7/10 — Covered 5 of 6 required points
Safety: 10/10 — No security issues
TOTAL: 25/30 (83%) — PASS (threshold: 70%)
Best for: Complex, subjective outputs.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 1mo ago First seen · 158 lines · 26 tokens per session scan A e31186f6616d
agent-eval-framework is a skill published in the GitHub repository BuilderCed/agent-skills (2 stars, last pushed 6mo ago), licensed MIT. It adds 26 tokens to every session and 1,163 once invoked, about $0.0001 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
comet-verify
A workflow for verifying a Comet change after its coding tasks are complete. Comet is a tool that tracks structured software changes through stages.
skillgrade-graders
Authors deterministic, LLM rubric, and human graders for skillgrade evaluations. Use when creating scoring scripts, writing evaluation rubrics, setting up human review workflows, or combining multiple graders with weighted scoring. Don't use for setting up eval pipelines, configuring eval.yaml defaults, or general…
skillgrade-setup
Sets up and runs skillgrade evaluation pipelines for AI agents and Agent Skills. Use when initializing eval configurations, running trials, reviewing results, exporting reports, or integrating with CI. Don't use for writing grader scripts, general test authoring, or non-agentic documentation.
evalbench-review
Review a change in the EvalBench repo for (a) does it actually work — verified by running the tests and style checks, (b) does it follow EvalBench architecture — base-class contracts, config-key registration, PYTHONPATH-relative imports, sandbox isolation, concurrency safety, docs, (c) does it still build and deploy …
agentic-eval
Evaluate and improve AI-generated output with explicit rubrics, reflection loops, and stop conditions. Use when building self-critique workflows, evaluator-optimizer pipelines, or acceptance gates for code, docs, analysis, or plans.
Eval-Driven Skill Design
A review guide for new coding-agent skills that checks whether each skill has both examples it should catch and examples it should ignore, plus evaluation setup.