Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add krzysztofsurdy/code-virtuoso --skill llm-evalsgit clone --depth 1 https://github.com/krzysztofsurdy/code-virtuosoWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/krzysztofsurdy/code-virtuoso/llm-evals)<a href="https://agentmods.dev/skills/krzysztofsurdy/code-virtuoso/llm-evals"><img src="https://agentmods.dev/badge/skills/krzysztofsurdy/code-virtuoso/llm-evals/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/krzysztofsurdy/code-virtuoso/llm-evals"><img src="https://agentmods.dev/badge/skills/krzysztofsurdy/code-virtuoso/llm-evals.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00051 | $0.02297 |
| Opus 5.5 | $0.00020 | $0.00919 |
| Sonnet 5.5 | $0.00010 | $0.00459 |
| Haiku 4.5 | $0.00005 | $0.00230 |
Grade A, and why
llm-evals scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured yesterday.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 134 lines — stays where its author put it; the contents beside it link to each section on GitHub.
LLM evals
An eval is a test for behavior you cannot specify with an exact expected output. The method is the same as any good test practice: look at real failures, write a check for each one, run the checks on every change. Most teams skip the first step and start with a dashboard of generic scores. Those scores look precise and tell you nothing about what breaks for your users.
Core Principles
| Principle | Meaning |
|---|---|
| Look at data first | Read real traces before writing any grader. The failure list comes from the data, not from a metrics catalog. |
| Binary verdicts | Each grader answers pass or fail for one specific failure. Scales hide disagreement and invite the middle score. |
| Code before models | Use an assertion when the property is checkable in code. A model judge is a last resort for what code cannot see. |
| One judge per failure | A judge that scores "overall quality" cannot be validated. A judge for one named failure can. |
| Validate the grader | An unvalidated LLM judge is an opinion. Measure it against human labels before trusting its number. |
| Grade outcomes | Check the end state of the world, not only what the agent said or the path it took. |
| Evals are code | Version them, run them in CI, and treat a drop as a failing test. |
Workflow
- Collect traces. Use production logs if you have them. Before launch, write realistic inputs by hand or generate varied ones and run them through the system. Capture full transcripts: inputs, tool calls, outputs. See error-analysis.
- Read and annotate. Open-code roughly 30 or more traces yourself, noting the first thing that went wrong in each. Continue toward about 100 diverse traces until new ones reveal no new kinds of failure.
- Build a failure taxonomy. Group the notes into named categories and count them. Fix the cheap ones first: a missing instruction or a broken tool is a bug, not an eval target.
- Pick a grader per remaining failure. Code assertion if possible, otherwise one validated judge. See the table below.
- Validate every judge against human labels with a train, dev, and test split. Report TPR and TNR on the held-out set. See judge-calibration.
- Assemble the regression suite. Real failures that you fixed, plus a few critical happy paths. Start with 20 to 50 cases, not hundreds. Keep it cheap enough to run on every change.
- Gate changes in CI with thresholds. Re-run error analysis on fresh traces every few weeks and after major changes, because the failure list drifts.
What ships with it
4 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- yesterday First seen · 134 lines · 51 tokens per session scan A b443a455081a
llm-evals is a skill published in the GitHub repository krzysztofsurdy/code-virtuoso (36 stars, last pushed today), licensed MIT. It adds 51 tokens to every session and 2,297 once invoked, about $0.0002 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-10-22.
Other skills, from other repositories
ml-data-leakage-guard
Detects and prevents data leakage in machine learning and mathematical modeling. Use after ML tasks involving data cleaning, feature engineering, data augmentation, algorithm development, normalization, missing value imputation, dimensionality reduction, feature selection, or time series modeling. Checks if…
coverage-tracker
Run a Google Alerts-style keyword coverage tracker. Uses news-search for recent keyword queries, lets the LLM dedupe and classify real features versus junk, stores decisions in SQLite, and alerts only on new real coverage.
prompt-set-qa
Gate a prompt universe for schema and provenance completeness, target or campaign contamination, evidence entailment, naturalness, one-concept clarity, architecture consistency, aided status, answer leakage, and semantic duplicates. Use after realistic prompt generation and before human panel selection.
realistic-prompt-generation
Generate natural, controlled prompt variants from a target-blind design brief and prompt architecture while preserving approved jobs, acts, journeys, constraints, roles, locales, proximity bands, and evidence language. Use after architecture design and before contamination or semantic QA.
ai-visibility-panel-design
Select QA-approved canonical intent cells into a versioned AI-visibility tracking panel with partitions, variants, lanes, surfaces, locales, repetitions, separate exposure and priority weights, randomization, uncertainty, refresh rules, and campaign controls. Use after prompt QA or when revising an existing panel.
prompt-regression-suite
Design a regression test suite that catches an LLM feature getting worse when the prompt, model, or context changes. Use when asked to stop prompt changes breaking production, set up golden tests or CI gates for an LLM feature, or test a model/prompt upgrade before shipping it. Produces a golden case set, per-case…