Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add Notysoty/openagentskills --skill agent-eval-framework-buildergit clone --depth 1 https://github.com/Notysoty/openagentskillsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/notysoty/openagentskills/agent-eval-framework-builder)<a href="https://agentmods.dev/skills/notysoty/openagentskills/agent-eval-framework-builder"><img src="https://agentmods.dev/badge/skills/notysoty/openagentskills/agent-eval-framework-builder/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/notysoty/openagentskills/agent-eval-framework-builder"><img src="https://agentmods.dev/badge/skills/notysoty/openagentskills/agent-eval-framework-builder.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00036 | $0.01778 |
| Opus 5.5 | $0.00014 | $0.00711 |
| Sonnet 5.5 | $0.00007 | $0.00356 |
| Haiku 4.5 | $0.00004 | $0.00178 |
Grade A, and why
Agent Evaluation Framework Builder scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 225 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Agent Evaluation Framework Builder
What this skill does
This skill designs an evaluation framework for an LLM agent or pipeline. Most teams skip evals until something breaks in production — this skill helps you build evals before launch so you have a baseline, catch regressions, and measure quality improvements objectively. It covers dataset construction, metric selection, LLM-as-judge setup, and CI integration.
How to use
Claude Code / Cline
Copy this file to .agents/skills/agent-eval-framework-builder/SKILL.md in your project root.
Then ask:
- "Use the Agent Eval Framework Builder to design evals for our support chatbot."
- "Build an evaluation suite for our RAG pipeline."
Provide:
- What the agent does
- What "good output" looks like
- Sample inputs (5–10 examples if available)
- Whether you have ground-truth answers or need to generate them
Cursor / Codex
Describe the agent and its task alongside these instructions.
The Prompt / Instructions for the Agent
When asked to build an evaluation framework, produce the following:
Step 1 — Choose the right eval type
| Agent Task | Eval Type | Reason |
|---|---|---|
| Factual Q&A with known answers | Exact match / F1 | Ground truth available |
| Summarization, drafting | LLM-as-judge | No single right answer |
| Code generation | Unit test execution | Correctness is verifiable |
| Multi-step agent task | Trajectory scoring | Need to evaluate the path, not just the endpoint |
| Classification / routing | Accuracy, F1 | Categorical output |
| RAG retrieval | Recall@K, MRR | Measure retrieval quality separately |
Use multiple eval types for complex agents: trajectory scoring + LLM-as-judge output quality.
Step 2 — Build the evaluation dataset
Minimum viable eval dataset: 50 examples covering:
- 40% typical cases (what users actually ask)
- 30% edge cases (ambiguous, multi-part, or unusual queries)
- 20% adversarial cases (jailbreak attempts, out-of-scope requests)
- 10% regression cases (bugs you've fixed in the past)
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 1mo ago First seen · 225 lines · 36 tokens per session scan A 1c060cf32730
Agent Evaluation Framework Builder is a skill published in the GitHub repository Notysoty/openagentskills (9 stars, last pushed 2mo ago), licensed MIT. It adds 36 tokens to every session and 1,778 once invoked, about $0.0001 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
review-ugc-render
Mandatory pre-publish review gate for a UGC video render. Transcribes the finished render's AUDIO with Whisper and word-diffs it against the approved spoken script, then gates pinning the final render (videoprojectupsert patch.finalrenderid) — blocking a render whose generated audio mis-voices a word (e.g. the…
api-tester
Test and document API endpoints - validate responses, check status, generate examples.
visual-qa
Runs rigorous visual QA across web, terminal, and paginated surfaces: an Apple HIG-based checklist, paired light/dark captures at phone and desktop widths, screenshot evidence, and a per-item PASS/FAIL verdict. Use for any UI build or change, or when asked whether a page, component, or TUI looks right or follows the…
remove-ai-slops
Removes AI-generated code smells from branch changes or an explicit file list behind regression tests. Use when the user asks to clean up, deslop, or remove AI-slop patterns from recent changes.
senpi-qa
QA the omo Senpi adapter (packages/omo-senpi, packages/senpi-task) against the REAL senpi binary in strict isolation, and write every artifact to the one canonical evidence path .omo/evidence/omo-senpi-adapter/ /. The live drivers under packages/omo-senpi/scripts/qa/ create their own isolated SENPICODINGAGENTDIR and…
qa
Exercise the delivered customer journey using real runtime or browser evidence. Use for functional acceptance testing after implementation, including failure paths.