Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add sananthanarayan/skilldrop --skill llm-eval-harnessgit clone --depth 1 https://github.com/sananthanarayan/skilldropWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/sananthanarayan/skilldrop/llm-eval-harness)<a href="https://agentmods.dev/skills/sananthanarayan/skilldrop/llm-eval-harness"><img src="https://agentmods.dev/badge/skills/sananthanarayan/skilldrop/llm-eval-harness/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/sananthanarayan/skilldrop/llm-eval-harness"><img src="https://agentmods.dev/badge/skills/sananthanarayan/skilldrop/llm-eval-harness.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00106 | $0.01882 |
| Opus 5 | $0.00053 | $0.00941 |
| Sonnet 5 | $0.00021 | $0.00376 |
| Haiku 4.5 | $0.00011 | $0.00188 |
Grade A, and why
llm-eval-harness scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 8d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 74 lines — stays where its author put it; the contents beside it link to each section on GitHub.
llm-eval-harness
Builds the measurement that turns "the new prompt feels better" into "the new prompt scores 0.91 vs 0.84 baseline, with zero regressions on the critical subset." Without it, every prompt or model change is a vibe with a deploy button. Provider-neutral by design — the harness shape is the same whether the feature runs on Claude, GPT, Gemini, or a local model; only the runner differs. Distinct from ai-usage-report (telemetry after the fact) and success-metrics (product outcomes) — this is the dev-loop quality gate.
How to respond
-
Pin the task and the unit of judgment. What does the feature do (classify / extract / summarize / answer-with-RAG / agentic-multi-step), and what does one gradeable output look like? Ask at most 2 questions, spent on the failure that hurts most ("what's a wrong answer that would actually cause a problem?") and whether ground truth exists. The answer-that-hurts shapes the adversarial cases and the critical subset.
-
Build the golden set with three deliberate buckets (case format in
templates/):- Representative — the real distribution of inputs, sampled from production/logs where possible, not invented. This sets the headline number.
- Adversarial / edge — the inputs that break things: ambiguous, out-of-scope, prompt-injection attempts, empty/malformed, the long tail. This is where models actually differ.
- Regression — every past failure, frozen as a case the moment it's fixed, so it can never silently return. Size honestly: 50 real, well-labeled cases beat 5,000 synthetic ones. State the count per bucket and how cases were sourced; a golden set of model-generated inputs graded by a model is a hall of mirrors, not an eval.
-
Choose the cheapest adequate grading method per case (decision tree in
reference.md) — descending order of preference, because cheaper means faster, deterministic, and trustworthy:- Programmatic / exact — string/JSON match, regex, schema validation, numeric tolerance. Use wherever the output is checkable. Free and non-negotiable when applicable.
- Structured assertions — "contains X", "cites a real source from the context", "valid JSON with field Y in range". Deterministic checks on unstructured output.
- LLM-as-judge — only when quality is genuinely subjective (helpfulness, tone, faithfulness). And when used, the judge gets its own rubric, its own validation against human labels, and controls for its known biases (position, verbosity, self-preference). An unvalidated judge is an opinion you've automated.
What ships with it
4 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 8d ago First seen · 74 lines · 106 tokens per session scan A 30c29d84f42e
llm-eval-harness is a skill published in the GitHub repository sananthanarayan/skilldrop (2 stars, last pushed 24d ago), licensed MIT. It adds 106 tokens to every session and 1,882 once invoked, about $0.0005 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
evaluating-llms-harness
Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.
cli-eval
Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.
data-quality-frameworks
Implement data quality validation with Great Expectations, dbt tests, and data contracts. Use when building data quality pipelines, implementing validation rules, or establishing data contracts.
model-merging
Merge multiple fine-tuned models using mergekit to combine capabilities without retraining. Use when creating specialized models by blending domain-specific expertise (math + coding + chat), improving performance beyond single models, or experimenting rapidly with model variants. Covers SLERP, TIES-Merging, DARE, Task…
dbt-transformation-patterns
Master dbt (data build tool) for analytics engineering with model organization, testing, documentation, and incremental strategies. Use when building data transformations, creating data models, or implementing analytics engineering best practices.
darwinian-evolver
Evolve prompts/regex/SQL/code with Imbue's evolution loop.