Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add liqiongyu/lenny_skills_plus --skill ai-evalsgit clone --depth 1 https://github.com/liqiongyu/lenny_skills_plusWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/liqiongyu/lenny_skills_plus/ai-evals)<a href="https://agentmods.dev/skills/liqiongyu/lenny_skills_plus/ai-evals"><img src="https://agentmods.dev/badge/skills/liqiongyu/lenny_skills_plus/ai-evals/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/liqiongyu/lenny_skills_plus/ai-evals"><img src="https://agentmods.dev/badge/skills/liqiongyu/lenny_skills_plus/ai-evals.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00046 | $0.02076 |
| Opus 5 | $0.00023 | $0.01038 |
| Sonnet 5 | $0.00009 | $0.00415 |
| Haiku 4.5 | $0.00005 | $0.00208 |
Grade A, and why
ai-evals scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 12d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 125 lines — stays where its author put it; the contents beside it link to each section on GitHub.
AI Evals
Scope
Covers
- Designing evaluation (“evals”) for LLM/AI features as an execution contract: what “good” means and how it’s measured
- Converting failures into a golden test set + error taxonomy + rubric
- Choosing a judging approach (human, LLM-as-judge, automated checks) and a repeatable harness/runbook
- Producing decision-ready results and an iteration loop (every bug becomes a new test)
When to use
- “Design evals for this LLM feature so we can ship with confidence.”
- “Create a rubric + golden set + benchmark for our AI assistant/copilot.”
- “We’re seeing flaky quality—do error analysis and turn it into a repeatable eval.”
- “Compare prompts/models safely with a clear acceptance threshold.”
When NOT to use
- You need to decide what to build (use
problem-definition,building-with-llms, orai-product-strategy). - You’re primarily doing traditional non-LLM software testing (use your standard eng QA/unit/integration tests).
- You want model training research or infra design (this skill assumes API/model usage; delegate to ML/infra).
- You only want vendor/model selection with no defined task + data (use
evaluating-new-technologyfirst, then come back with a concrete use case). - You want to measure overall product-market fit or retention, not AI output quality (use
measuring-product-market-fit). - You need a product requirements document that includes but goes beyond eval design (use
writing-prds).
Inputs
Minimum required
- System under test (SUT): what the AI does, for whom, in what workflow (inputs → outputs)
- The decision the eval must support (ship/no-ship, compare options, regression gate)
- What “good” means: 3–10 target behaviors + top failure modes
- Constraints: privacy/compliance, safety policy, languages, cost/latency budgets, timeline
Missing-info strategy
- Ask up to 5 questions from references/INTAKE.md (3–5 at a time).
- If details remain missing, proceed with explicit assumptions and provide 2–3 viable options (judge type, scoring scheme, dataset size).
- If asked to run code or generate datasets from sensitive sources, request confirmation and apply least privilege (no secrets; redact/anonymize).
What ships with it
13 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
- eval/eval_config.json 655 B
- eval/SHOWCASE.md 4.0 KB
- eval/with_skill.md 61 KB
- eval/without_skill.md 22 KB
- README.md 1.6 KB
- references/CHECKLISTS.md 2.1 KB
- references/EXAMPLES.md 1.7 KB
- references/INTAKE.md 1.7 KB
- references/RUBRIC.md 3.0 KB
- references/SOURCE_SUMMARY.md 1018 B
- references/TEMPLATES.md 3.9 KB
- references/WORKFLOW.md 3.0 KB
- skillpack.json 345 B
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 12d ago First seen · 125 lines · 46 tokens per session scan A 6327b078e7c0
ai-evals is a skill published in the GitHub repository liqiongyu/lenny_skills_plus (52 stars, last pushed 3mo ago), licensed Apache-2.0. It adds 46 tokens to every session and 2,076 once invoked, about $0.0002 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
plan
Use when a request needs shaping before any code is written — a rough or vague prompt to sharpen, an ambiguous idea to design, or a clear-enough task to decompose. One chain-starter that amplifies the prompt, designs the approach, and decomposes it into a batched task file, skipping whichever phases the request…
config-evals
Builds and maintains configuration-based evaluations on a workflow with the eval-config tool. Use when the user asks to set up, add, view, change, or remove an evaluation, score, grade, or judge a workflow's output, or measure answer quality against a test dataset. This is the only eval form Instance AI handles — it…
recipe-eval-prompt
Compares original and optimized prompts through repeated blind paired execution in git worktrees. Use when evaluating prompt improvement effects or learning prompt engineering through concrete examples.
test-agent
Systematisches Testen von BACH und anderen LLM-OS-Systemen. Orchestrator fuer Test- und Vergleichs-Workflows.
eval-engine
Use this skill when a user supplies an AI feature spec, PRD, PMOS/task contract, existing eval suite, traces, outputs, state evidence, or release question and wants to define good, create or run evals, grade outcomes, agent trajectories, end-to-end system checkpoints, or promised memory behavior, add deterministic or…
eval-before-ship
Use before merging, deploying, or demo'ing any LLM feature. Requires documented eval results — pass rate, failure analysis, baseline comparison. Blocks "it looked good when I tested it" completions.