Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/ancoleman/ai-design-components/evaluating-llmsnpx skills add ancoleman/ai-design-components --skill evaluating-llmsgit clone --depth 1 https://github.com/ancoleman/ai-design-componentsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/ancoleman/ai-design-components/evaluating-llms)<a href="https://agentmods.dev/skills/ancoleman/ai-design-components/evaluating-llms"><img src="https://agentmods.dev/badge/skills/ancoleman/ai-design-components/evaluating-llms.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00050 | $0.04422 |
| Opus 5 | $0.00025 | $0.02211 |
| Sonnet 5 | $0.00010 | $0.00884 |
| Haiku 4.5 | $0.00005 | $0.00442 |
Grade A, and why
evaluating-llms scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 5d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 486 lines — stays where its author put it; the contents beside it link to each section on GitHub.
LLM Evaluation
Evaluate Large Language Model (LLM) systems using automated metrics, LLM-as-judge patterns, and standardized benchmarks to ensure production quality and safety.
When to Use This Skill
Apply this skill when:
- Testing individual prompts for correctness and formatting
- Validating RAG (Retrieval-Augmented Generation) pipeline quality
- Measuring hallucinations, bias, or toxicity in LLM outputs
- Comparing different models or prompt configurations (A/B testing)
- Running benchmark tests (MMLU, HumanEval) to assess model capabilities
- Setting up production monitoring for LLM applications
- Integrating LLM quality checks into CI/CD pipelines
Common triggers:
- "How do I test if my RAG system is working correctly?"
- "How can I measure hallucinations in LLM outputs?"
- "What metrics should I use to evaluate generation quality?"
- "How do I compare GPT-4 vs Claude for my use case?"
- "How do I detect bias in LLM responses?"
Evaluation Strategy Selection
Decision Framework: Which Evaluation Approach?
By Task Type:
| Task Type | Primary Approach | Metrics | Tools |
|---|---|---|---|
| Classification (sentiment, intent) | Automated metrics | Accuracy, Precision, Recall, F1 | scikit-learn |
| Generation (summaries, creative text) | LLM-as-judge + automated | BLEU, ROUGE, BERTScore, Quality rubric | GPT-4/Claude for judging |
| Question Answering | Exact match + semantic similarity | EM, F1, Cosine similarity | Custom evaluators |
| RAG Systems | RAGAS framework | Faithfulness, Answer/Context relevance | RAGAS library |
| Code Generation | Unit tests + execution | Pass@K, Test pass rate | HumanEval, pytest |
| Multi-step Agents | Task completion + tool accuracy | Success rate, Efficiency | Custom evaluators |
By Volume and Cost:
| Samples | Speed | Cost | Recommended Approach |
|---|---|---|---|
| 1,000+ | Immediate | $0 | Automated metrics (regex, JSON validation) |
| 100-1,000 | Minutes | $0.01-0.10 each | LLM-as-judge (GPT-4, Claude) |
| < 100 | Hours | $1-10 each | Human evaluation (pairwise comparison) |
What ships with it
17 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
- examples/python/benchmark_testing.py 16 KB runs code
- examples/python/classification_metrics.py 14 KB runs code
- examples/python/deepeval_example.py 14 KB runs code
- examples/python/llm_as_judge.py 17 KB runs code
- examples/python/ragas_example.py 11 KB runs code
- examples/python/unit_evaluation.py 13 KB runs code
- examples/typescript/langsmith-integration.ts 12 KB runs code
- examples/typescript/llm-as-judge.ts 13 KB runs code
- examples/typescript/unit-evaluation.ts 13 KB runs code
- outputs.yaml 8.0 KB
- references/benchmarks.md 15 KB
- references/evaluation-types.md 12 KB
- references/llm-as-judge.md 12 KB
- references/metrics-reference.md 15 KB
- references/production-evaluation.md 19 KB
- references/rag-evaluation.md 18 KB
- references/safety-evaluation.md 20 KB
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 5d ago First seen · 486 lines · 50 tokens per session scan A 55f8ebd7adde
evaluating-llms is a skill published in the GitHub repository ancoleman/ai-design-components (518 stars, last pushed 8mo ago), licensed MIT. It adds 50 tokens to every session and 4,422 once invoked, about $0.0003 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
ai-feedback-loops
Design feedback mechanisms that help AI systems learn from users - thumbs up/down, preference ranking, corrections, and human-in-the-loop escalation. Use when: RLHF UX, user feedback for AI, thumbs up down design, AI correction flow, human in the loop, feedback signal design, AI improvement loops.
ai-error-resilience
Design graceful failure experiences for AI products - hallucinations, uncertainty, wrong outputs, and edge cases. Use when: AI hallucination UX, error handling for AI, uncertainty design, graceful degradation, AI failure recovery, confidence thresholds, safe fallbacks.
ai-prompt-ux
Design the input experience for AI products - how users craft, structure, and refine their instructions to AI systems. Use when: prompt interface, AI input design, prompt templates, prompt suggestions, context window UX, instruction design, AI input affordances, prompt engineering UX.
ai-safety-guardrails
Design safety experiences for AI products - content moderation UX, bias detection surfaces, harm prevention patterns, and responsible AI interfaces. Use when: AI safety UX, content moderation, responsible AI, AI bias UX, harm prevention, content filtering UX, AI refusal design, safety disclaimers.
ai-multimodal-output
Design how AI presents results across text, code, images, charts, and mixed media - response formatting, output hierarchy, and cross-modal transitions. Use when: AI output design, response formatting, AI results display, multimodal output, AI-generated content presentation, code output UX, AI visualization.
ai-trust-transparency
Design explainability interfaces that help users understand AI decisions, build calibrated trust, and verify AI outputs. Use when: AI explainability, XAI UX, confidence indicators, citation design, source attribution, trust signals, AI transparency, why did AI do this.