Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add di37/EvalSurfer --skill eval-surfergit clone --depth 1 https://github.com/di37/EvalSurferWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/di37/evalsurfer/eval-surfer)<a href="https://agentmods.dev/skills/di37/evalsurfer/eval-surfer"><img src="https://agentmods.dev/badge/skills/di37/evalsurfer/eval-surfer.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00081 | $0.02203 |
| Opus 5 | $0.00041 | $0.01102 |
| Sonnet 5 | $0.00016 | $0.00441 |
| Haiku 4.5 | $0.00008 | $0.00220 |
Grade A, and why
eval-surfer scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 6d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 166 lines — stays where its author put it; the contents beside it link to each section on GitHub.
EvalSurfer
EvalSurfer is a skill-first evaluation workflow that implements the CIMAA framework — Core, Interface, Metrics, Analysis, Assurance. The skill drives the assessment; the framework definitions, Python functions, and CLI are supporting utilities.
Use This Skill When
- The user asks to evaluate an AI application, answer, RAG pipeline, agent, chatbot, or production LLM workflow.
- The user needs a rubric, scorecard, benchmark, eval report, or quality gate.
- The task involves correctness, relevance, completeness, instruction following, groundedness, citation accuracy, tool use, multi-turn behavior, safety, latency, TTFT, cost, or reliability.
Workflow
- Clarify the evaluation target: answer, conversation, RAG run, agent trace, or production logs.
- Scope adaptively. Infer which categories and criteria apply from the inputs that are actually present instead of evaluating everything. The planner decides this deterministically:
echo '{"sample": {"query": "...", "answer": "...", "retrieved_docs": ["..."]}}' | python -m evalsurfer.interface.cli.plan - --pretty
It returns the applicable criteria (with a reason for each skip) and a coverage score. The three rubric categories use report section keys/nesting, not CIMAA layer nesting or ownership:
- Quality (agent-judged; nested under
report.metrics.quality): correctness, relevance, completeness, instruction following, RAG quality, tool use, and multi-turn behavior. Distinct from Metrics reference metrics (BLEU / ROUGE / …). - Operational (Metrics layer; auto-scored from traces): the five numbers of inference — TTFT, inter-token latency, throughput (TPS), P99 tail latency, and $/1M tokens — plus end-to-end and under-load latency, cost per request, token efficiency, and failure rate.
- Safety (Assurance layer; agent-judged + red-team): toxicity, harmful content, bias/fairness, PII leakage, and jailbreak resistance — assessed by default; opt out only for low-risk targets, and record why.
- Assess only the applicable criteria. Record each skipped criterion with the planner's reason under
Not assessed; never guess. - Use
spec/framework.yamlorspec/framework.jsonfor the machine-readable rubric. - If operational traces are provided, calculate metrics with the CLI:
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 6d ago First seen · 166 lines · 81 tokens per session scan A c5f2d49dd99a
eval-surfer is a skill published in the GitHub repository di37/EvalSurfer (11 stars, last pushed 1mo ago), licensed MIT. It adds 81 tokens to every session and 2,203 once invoked, about $0.0004 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
Evaluation
Frames model, prompt, and system evaluation as a reproducible experiment with baselines, datasets, and explicit metrics.
ai-engineering-toolkit
6 production-ready AI engineering workflows: prompt evaluation (8-dimension scoring), context budget planning, RAG pipeline design, agent security audit (65-point checklist), eval harness building, and product sense coaching.
ai-engineering-toolkit
6 production-ready AI engineering workflows: prompt evaluation (8-dimension scoring), context budget planning, RAG pipeline design, agent security audit (65-point checklist), eval harness building, and product sense coaching.
connect-agent
Connect the codebase's AI agent to LangWatch agent simulations, so test suites run against the real agent process. Adds a small connect function beside the service startup that calls the agent already in the codebase, which opens an outbound connection and registers the agent with its environment and its run…
prompt-optimization
Improve a prompt on the evaluations workbench through a measured loop. Score the baseline first, then duplicate the target column, form a hypothesis from failing rows, edit the copy's prompt draft, run, compare pass rate and cost, and repeat until the numbers hold. Use when the user asks to optimize or improve a…
debug-with-langwatch
Root-cause production errors and misbehaving agent runs with LangWatch. Finds errored traces, inspects spans, checks monitor and evaluator scores, then narrows to a root cause. Use when something is failing or misbehaving in production (errors, bad answers, latency spikes).