Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/phuonghx/aim-cli/llm-evaluationnpx skills add phuonghx/aim-cli --skill llm-evaluationgit clone --depth 1 https://github.com/phuonghx/aim-cliWhat it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00095 | $0.01523 |
| Opus 5 | $0.00048 | $0.00762 |
| Sonnet 5 | $0.00019 | $0.00305 |
| Haiku 4.5 | $0.00010 | $0.00152 |
Grade A, and why
llm-evaluation scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured yesterday.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 129 lines — stays where its author put it; the contents beside it link to each section on GitHub.
LLM Evaluation
If you cannot measure it, you are tuning prompts in the dark. Build the eval before you tune.
When to add evals
- Before tuning a prompt or swapping models — you need a baseline.
- When shipping any LLM feature that real users or systems depend on.
- After every production bug — turn the failing input into a permanent test case.
- When "it feels better" is the only evidence a change is an improvement.
Skip formal evals only for throwaway scripts. Anything that ships gets at least a small fixed set.
Build a golden dataset
A golden set is a fixed collection of (input, expected) pairs that defines "correct" for your task.
- Start small, start real. 20–50 hand-curated cases beat 5,000 synthetic ones. Grow over time.
- Cover the distribution: common cases, edge cases, the null/empty case, and known past failures.
- Freeze it. The set must be stable so scores are comparable across runs. Version it (
eval/v2). - Label deliberately. Each
expectedshould be defensible; ambiguous labels poison every metric. - Keep it in the repo, reviewed like code.
Avoid leakage
Leakage = your eval secretly rewards memorization or itself, inflating scores.
- Do not put eval examples into the prompt's few-shot block. Test and demonstration sets must be disjoint.
- Do not tune the prompt by staring at eval answers — tune on a separate dev split, report on a held-out split.
- Watch for train/test contamination when inputs come from public data the model may have seen.
- If you use an LLM judge, the judge should not be the same call that produced the answer.
Metrics: pick by task
| Metric | Use for | How |
|---|---|---|
| Exact / normalized match | Classification, extraction, enums | Compare after normalizing case/whitespace |
| Field-level / JSON match | Structured output | Compare per key; report which fields fail |
| Rubric (LLM-as-judge) | Open-ended generation, summaries | Score against an explicit rubric, 1–5 |
| pass@k | Code / tasks with a verifier | Sample k; pass if any sample passes the check |
| Task success | Agents / tool use | Did it reach the goal state? (programmatic check) |
| Regression rate | Any fixed set | % of previously-passing cases now failing |
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- yesterday First seen · 129 lines · 95 tokens per session scan A bed81fb52e43
llm-evaluation is a skill published in the GitHub repository phuonghx/aim-cli (1 stars, last pushed 2mo ago), licensed MIT. It adds 95 tokens to every session and 1,523 once invoked, about $0.0005 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
hs-release
Cut a core Hindsight release (vX.Y.Z) and open the changelog + blog PR. Use when asked to cut/start a release, bump the version, or publish a new Hindsight version.
hindsight-local
Store user preferences, learnings from tasks, and procedure outcomes. Use to remember what works and recall context before new tasks. (user).
research-repository
Build a repository that makes findings findable, reusable, and cumulative across teams. Use when the same research keeps getting redone. For synthesising one study, use affinity-diagram.
design-negotiation
Advocate for design quality, scope, and timeline with partners and leadership using evidence and shared goals. Use in the conversation itself. For the commercial vocabulary behind it, use business-design (ux-strategy).
user-persona
Build research-grounded personas with goals, frustrations, and behavioural patterns. Use when decisions need a consistent user reference. For one session's emotional snapshot use empathy-map; for motivation framing use jobs-to-be-done.
version-control-strategy
Define version control for design files, components, and libraries — branching, naming, and release. Use when file history is chaotic. For design system contribution rules, use design-system-governance (design-systems).