Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/notysoty/openagentskills/agent-eval-framework-buildernpx skills add Notysoty/openagentskills --skill agent-eval-framework-buildergit clone --depth 1 https://github.com/Notysoty/openagentskillsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/notysoty/openagentskills/agent-eval-framework-builder)<a href="https://agentmods.dev/skills/notysoty/openagentskills/agent-eval-framework-builder"><img src="https://agentmods.dev/badge/skills/notysoty/openagentskills/agent-eval-framework-builder.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00036 | $0.01778 |
| Opus 5 | $0.00018 | $0.00889 |
| Sonnet 5 | $0.00007 | $0.00356 |
| Haiku 4.5 | $0.00004 | $0.00178 |
Grade A, and why
Agent Evaluation Framework Builder scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 4d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 225 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Agent Evaluation Framework Builder
What this skill does
This skill designs an evaluation framework for an LLM agent or pipeline. Most teams skip evals until something breaks in production — this skill helps you build evals before launch so you have a baseline, catch regressions, and measure quality improvements objectively. It covers dataset construction, metric selection, LLM-as-judge setup, and CI integration.
How to use
Claude Code / Cline
Copy this file to .agents/skills/agent-eval-framework-builder/SKILL.md in your project root.
Then ask:
- "Use the Agent Eval Framework Builder to design evals for our support chatbot."
- "Build an evaluation suite for our RAG pipeline."
Provide:
- What the agent does
- What "good output" looks like
- Sample inputs (5–10 examples if available)
- Whether you have ground-truth answers or need to generate them
Cursor / Codex
Describe the agent and its task alongside these instructions.
The Prompt / Instructions for the Agent
When asked to build an evaluation framework, produce the following:
Step 1 — Choose the right eval type
| Agent Task | Eval Type | Reason |
|---|---|---|
| Factual Q&A with known answers | Exact match / F1 | Ground truth available |
| Summarization, drafting | LLM-as-judge | No single right answer |
| Code generation | Unit test execution | Correctness is verifiable |
| Multi-step agent task | Trajectory scoring | Need to evaluate the path, not just the endpoint |
| Classification / routing | Accuracy, F1 | Categorical output |
| RAG retrieval | Recall@K, MRR | Measure retrieval quality separately |
Use multiple eval types for complex agents: trajectory scoring + LLM-as-judge output quality.
Step 2 — Build the evaluation dataset
Minimum viable eval dataset: 50 examples covering:
- 40% typical cases (what users actually ask)
- 30% edge cases (ambiguous, multi-part, or unusual queries)
- 20% adversarial cases (jailbreak attempts, out-of-scope requests)
- 10% regression cases (bugs you've fixed in the past)
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 4d ago First seen · 225 lines · 36 tokens per session scan A 1c060cf32730
Agent Evaluation Framework Builder is a skill published in the GitHub repository Notysoty/openagentskills (9 stars, last pushed 20d ago), licensed MIT. It adds 36 tokens to every session and 1,778 once invoked, about $0.0002 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
seedance-antislop
Detect and remove hollow AI filler language, empty superlatives, and vague boosters that degrade Seedance 2.0 prompt quality. Use when a prompt feels generic, over-written, or 'AI-sounding', or when generation output looks bland and needs a quality pass.
red-green-refactor
Guides the red-green-refactor TDD workflow: write a failing test first, implement the minimum code to make it pass, then refactor while keeping tests green. Use when a user asks to practice TDD, write tests first, follow red-green-refactor, do test-driven development, write failing tests before code, or phrases like…
AI System Quality Engineer
Test LLM, RAG, MCP, and agentic systems end to end. Build golden datasets, run deterministic checks and LLM judges, score retrieval, probe prompt injection, verify tool use, and gate CI on thresholds. Orchestrates DeepEval, Ragas, promptfoo, and Langfuse.
analyse-problem
Comprehensive A3 one-page problem analysis with root cause and action plan.
create-skill
Guide for creating effective skills. This command should be used when users want to create a new skill (or update an existing skill) that extends Claude's capabilities with specialized knowledge, workflows, or tool integrations. Use when creating new skills, editing existing skills, or verifying skills work before…
do-competitively
Execute tasks through competitive multi-agent generation, meta-judge evaluation specification, multi-judge evaluation, and evidence-based synthesis.