Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add phuonghx/aim-cli --skill llm-evaluationgit clone --depth 1 https://github.com/phuonghx/aim-cliWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/phuonghx/aim-cli/llm-evaluation)<a href="https://agentmods.dev/skills/phuonghx/aim-cli/llm-evaluation"><img src="https://agentmods.dev/badge/skills/phuonghx/aim-cli/llm-evaluation/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/phuonghx/aim-cli/llm-evaluation"><img src="https://agentmods.dev/badge/skills/phuonghx/aim-cli/llm-evaluation.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00095 | $0.01523 |
| Opus 5.5 | $0.00038 | $0.00609 |
| Sonnet 5.5 | $0.00019 | $0.00305 |
| Haiku 4.5 | $0.00010 | $0.00152 |
Grade A, and why
llm-evaluation scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 129 lines — stays where its author put it; the contents beside it link to each section on GitHub.
LLM Evaluation
If you cannot measure it, you are tuning prompts in the dark. Build the eval before you tune.
When to add evals
- Before tuning a prompt or swapping models — you need a baseline.
- When shipping any LLM feature that real users or systems depend on.
- After every production bug — turn the failing input into a permanent test case.
- When "it feels better" is the only evidence a change is an improvement.
Skip formal evals only for throwaway scripts. Anything that ships gets at least a small fixed set.
Build a golden dataset
A golden set is a fixed collection of (input, expected) pairs that defines "correct" for your task.
- Start small, start real. 20–50 hand-curated cases beat 5,000 synthetic ones. Grow over time.
- Cover the distribution: common cases, edge cases, the null/empty case, and known past failures.
- Freeze it. The set must be stable so scores are comparable across runs. Version it (
eval/v2). - Label deliberately. Each
expectedshould be defensible; ambiguous labels poison every metric. - Keep it in the repo, reviewed like code.
Avoid leakage
Leakage = your eval secretly rewards memorization or itself, inflating scores.
- Do not put eval examples into the prompt's few-shot block. Test and demonstration sets must be disjoint.
- Do not tune the prompt by staring at eval answers — tune on a separate dev split, report on a held-out split.
- Watch for train/test contamination when inputs come from public data the model may have seen.
- If you use an LLM judge, the judge should not be the same call that produced the answer.
Metrics: pick by task
| Metric | Use for | How |
|---|---|---|
| Exact / normalized match | Classification, extraction, enums | Compare after normalizing case/whitespace |
| Field-level / JSON match | Structured output | Compare per key; report which fields fail |
| Rubric (LLM-as-judge) | Open-ended generation, summaries | Score against an explicit rubric, 1–5 |
| pass@k | Code / tasks with a verifier | Sample k; pass if any sample passes the check |
| Task success | Agents / tool use | Did it reach the goal state? (programmatic check) |
| Regression rate | Any fixed set | % of previously-passing cases now failing |
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 1mo ago First seen · 129 lines · 95 tokens per session scan A bed81fb52e43
llm-evaluation is a skill published in the GitHub repository phuonghx/aim-cli (1 stars, last pushed 3mo ago), licensed MIT. It adds 95 tokens to every session and 1,523 once invoked, about $0.0004 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
ai-observability-promptfoo
Testing and evaluation framework for LLM prompts and applications -- promptfooconfig.yaml, assertions, model-graded evals, red teaming, CI/CD integration, custom providers, and comparative evaluation.
mcp-tool-developer
Build Model Context Protocol (MCP) servers and tools from scratch. Full-stack MCP development with TypeScript/Python, testing, deployment, and registry publishing.
adaline-evaluations
Run and manage evaluations in Adaline to test prompt quality at scale. Use when creating evaluation runs, polling status, analyzing results, or cancelling runs.
data-quality-frameworks
Implement data quality validation with Great Expectations, dbt tests, and data contracts. Use when building data quality pipelines, implementing validation rules, or establishing data contracts.
dbt-transformation-patterns
Master dbt (data build tool) for analytics engineering with model organization, testing, documentation, and incremental strategies. Use when building data transformations, creating data models, or implementing analytics engineering best practices.
nemo-automodel-model-onboarding
Guide for onboarding new model architectures into NeMo AutoModel, including architecture discovery, implementation patterns, registration, and validation.