Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add po4yka/llm-wiki-skills --skill llm-wiki-eval-toolinggit clone --depth 1 https://github.com/po4yka/llm-wiki-skillsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/po4yka/llm-wiki-skills/llm-wiki-eval-tooling)<a href="https://agentmods.dev/skills/po4yka/llm-wiki-skills/llm-wiki-eval-tooling"><img src="https://agentmods.dev/badge/skills/po4yka/llm-wiki-skills/llm-wiki-eval-tooling/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/po4yka/llm-wiki-skills/llm-wiki-eval-tooling"><img src="https://agentmods.dev/badge/skills/po4yka/llm-wiki-skills/llm-wiki-eval-tooling.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00071 | $0.02076 |
| Opus 5 | $0.00036 | $0.01038 |
| Sonnet 5 | $0.00014 | $0.00415 |
| Haiku 4.5 | $0.00007 | $0.00208 |
Grade A, and why
llm-wiki-eval-tooling scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 12d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 255 lines — stays where its author put it; the contents beside it link to each section on GitHub.
LLM-Wiki Eval Tooling
Goal
Turn LLM-Wiki usefulness, retrieval quality, grounding, security and maintenance questions into measurable tests, scorecards and repeatable CI/review gates.
Use references/docs/18-evaluation-methodology.md as the reference methodology for metric definitions, experiment design, dataset construction, CI strategy and rollout.
When to use
- The user asks which eval framework, dataset format, scorecard, or CI gate fits their LLM-Wiki (e.g. "should we use Ragas or promptfoo", "what does a retrieval eval set look like").
- The user wants to prove or disprove a claim such as "the wiki saves time" or "hybrid retrieval improved answers" with measurable tests.
- The user needs to design PR-time, nightly, or release-time CI gates for retrieval, grounding, or security regressions.
- The user needs a red-team/prompt-injection test plan or a human calibration workflow for LLM judges.
- Route to
llm-wiki-evalinstead when the user wants an actual pilot run executed, not tool/format selection.
Inputs
- Wiki path and evaluation scope.
- User claim to test, such as “the wiki saves time” or “hybrid retrieval improved answers”.
- Recent real questions/tasks or production traces.
- Retrieval layer details.
- Model/provider policy.
- Desired tools: Ragas, promptfoo, DeepEval, TruLens, LangSmith, OpenAI Evals/API Evals, Phoenix/Arize, pytrec_eval, custom scripts.
- Security constraints and risk tier.
Procedure
1. Pick evaluation layer
| Layer | Question | Tools/metrics |
|---|---|---|
| Retrieval | Did we find the right wiki/source pages? | recall@k, MRR, nDCG, qrels, hit/miss labels, pytrec_eval, Ragas context precision/recall. |
| Grounding | Is the answer supported by sources? | citation coverage, unsupported-claim rate, claim audit, Ragas/DeepEval faithfulness, TruLens groundedness. |
| Answer quality | Does the answer solve the task? | human rubric, pairwise preference, model-graded rubric, correctness/completeness/actionability. |
| Wiki usefulness | Did the wiki reduce work? | retrieval hit rate, answer reuse, read/write ratio, output beyond vault, time saved estimate. |
| Prompt/model regression | Did behavior change after prompt/model edits? | promptfoo, DeepEval, LangSmith evals, snapshot tests, PromptEval-style prompt variants. |
| Safety | Can malicious sources or prompts bypass policy? | promptfoo red-team, indirect prompt injection tests, PII/secret canaries, OWASP-aligned fixtures. |
| Operational health | Is the wiki alive and trusted? | wiki-lint, stale-page count, review backlog, provenance coverage, broken links. |
What ships with it
5 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 12d ago First seen · 255 lines · 71 tokens per session scan A 143986841b96
llm-wiki-eval-tooling is a skill published in the GitHub repository po4yka/llm-wiki-skills (3 stars, last pushed 19d ago), licensed MIT. It adds 71 tokens to every session and 2,076 once invoked, about $0.0004 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
test-agent
Systematisches Testen von BACH und anderen LLM-OS-Systemen. Orchestrator fuer Test- und Vergleichs-Workflows.
read-search-smart
Semantic search via Smart Connections embeddings. Returns ranked chunks with cosine scores and breadcrumbs. Use when the query is conceptual (meaning, not literal substring). EN triggers: "find notes about X", "what do I have on X", "semantic search for X", "find concepts related to X", "notes similar to ". FR…
knowledge-base-interop
Two-layer knowledge architecture — curate in LLM Wiki, search in Astra KB. Covers input classification, source-to-wiki ingestion, wiki-to-KB export (batch + realtime), semantic chunking, SAG extraction, health checks, and multi-format export.
lint
A health check for the LLM Wiki, a knowledge base about large language models. It looks for missing pages, broken links, conflicting or outdated content, missing fields, incomplete topic maps, weak bias checks, and an old core-context snapshot.
knowject-rag-eval
A skill for creating or reviewing evaluation cases for RAG, a system that searches stored documents before generating an answer, including the quality of its citations.
cli-eval
Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.