Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add VoDaiLocz/kilo-kit-mcp --skill llm-evalsgit clone --depth 1 https://github.com/VoDaiLocz/kilo-kit-mcpWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/vodailocz/kilo-kit-mcp/llm-evals)<a href="https://agentmods.dev/skills/vodailocz/kilo-kit-mcp/llm-evals"><img src="https://agentmods.dev/badge/skills/vodailocz/kilo-kit-mcp/llm-evals/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/vodailocz/kilo-kit-mcp/llm-evals"><img src="https://agentmods.dev/badge/skills/vodailocz/kilo-kit-mcp/llm-evals.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- NVIDIA SkillSpector pass
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00045 | $0.00904 |
| Opus 5.5 | $0.00018 | $0.00362 |
| Sonnet 5.5 | $0.00009 | $0.00181 |
| Haiku 4.5 | $0.00005 | $0.00090 |
Grade A, and why
llm-evals scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 67 lines — stays where its author put it; the contents beside it link to each section on GitHub.
LLM Evaluation & Validation Framework
Overview
The llm-evals skill provides a systematic framework for evaluating and monitoring Large Language Model (LLM) applications throughout the development lifecycle. It bridges the gap between ad-hoc testing and production-grade reliability by implementing rigorous evaluation pipelines, metric-driven gating, and dataset versioning.
When To Use
- Setting up automated evaluation pipelines for LLM chains.
- Developing or refining RAG (Retrieval-Augmented Generation) systems.
- Preparing LLM applications for production deployment.
- Investigating performance regressions in complex, multi-step agent trajectories.
- Defining custom rubrics for LLM-as-a-Judge scenarios.
Core Concepts
- Trajectory-Level Evaluation: Scoring intermediate reasoning steps, tool choices, and argument precision, moving beyond just final output metrics.
- LLM-as-a-Judge: Using capable models (e.g., GPT-4o, Claude 3.5 Sonnet) as automated judges to score outputs based on user-defined rubrics.
- Metric Tiers:
- Unit: Single prompt/completion pairs.
- Integration: Individual chains or multi-step logic.
- System: End-to-end user intent fulfillment evaluation.
- Golden Datasets: Version-controlled suites containing ground-truth, negative test cases, and edge cases to ensure consistent benchmarks.
- Calibration: Ensuring LLM judges align with human grading, including inter-rater reliability checks.
Workflow
- Dataset Curation: Create and version control evaluation suites using JSON/YAML formats, incorporating synthetic failure injections.
- Metric Definition: Define evaluation criteria (e.g., faithfulness, relevancy, G-Eval) using established frameworks.
- Execution: Run evaluations locally or in CI environments using tools like
promptfooordeepeval. - Analysis: Review evaluation reports to identify bottlenecks or high-error clusters.
- CI/CD Integration: Incorporate gating steps in deployment pipelines to automatically block commits failing established regression thresholds.
- Continuous Monitoring: Shift evaluation metrics into production monitoring (e.g., Langfuse) to track drift.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 1mo ago First seen · 67 lines · 45 tokens per session scan A 1c1e3957a0db
llm-evals is a skill published in the GitHub repository VoDaiLocz/kilo-kit-mcp (28 stars, last pushed 3d ago), licensed Apache-2.0. It adds 45 tokens to every session and 904 once invoked, about $0.0002 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
advanced-evaluation
Design LLM-as-judge rubrics, direct scores and pairwise comparisons. Use for judge models, evals, position bias and calibration against human labels.
agent-evals
Build automated evaluation suites for AI agents using golden datasets, rubrics, and regression gates. Use when shipping agent features, validating prompt changes, or gating deployments on quality.
llm-app-patterns
Production-ready patterns for building LLM applications, inspired by Dify and industry best practices.
llm-structured-output
Get reliable JSON, enums, and typed objects from LLMs using responseformat, tooluse, and schema-constrained decoding across OpenAI, Anthropic, and Google APIs.
evals-router
Use when evaluating LLM or RAG outputs: audit eval coverage, analyze failed traces, write binary judge prompts, validate judges against labels, generate targeted synthetic cases, evaluate retrieval quality, or plan review tooling. Do not use for ordinary software test implementation.
llm-json-contract-check
Validate AI-generated JSON against schema and business rules, separating refusals, truncation, parse failures, schema violations, and domain invariant breaches. Use before any tool call, DB write, or external side effect consumes model output.