Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add po4yka/llm-wiki-skills --skill llm-wiki-evalgit clone --depth 1 https://github.com/po4yka/llm-wiki-skillsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/po4yka/llm-wiki-skills/llm-wiki-eval)<a href="https://agentmods.dev/skills/po4yka/llm-wiki-skills/llm-wiki-eval"><img src="https://agentmods.dev/badge/skills/po4yka/llm-wiki-skills/llm-wiki-eval/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/po4yka/llm-wiki-skills/llm-wiki-eval"><img src="https://agentmods.dev/badge/skills/po4yka/llm-wiki-skills/llm-wiki-eval.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00064 | $0.02198 |
| Opus 5 | $0.00032 | $0.01099 |
| Sonnet 5 | $0.00013 | $0.00440 |
| Haiku 4.5 | $0.00006 | $0.00220 |
Grade A, and why
llm-wiki-eval scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 11d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 293 lines — stays where its author put it; the contents beside it link to each section on GitHub.
LLM-Wiki Eval
Goal
Measure whether the wiki improves real work instead of merely accumulating attractive notes.
Use references/docs/18-evaluation-methodology.md for the detailed methodology: retrieval metrics, grounding metrics, with-wiki experiments, human calibration, operational health, security gates and rollout roadmap.
Use references/benchmarks/pilot-questions.md and references/benchmarks/scoring-rubric.md when the user wants a bounded pilot benchmark with baseline and with-wiki passes.
When to use
- The user asks to measure, benchmark or score whether the wiki is actually useful, not just growing.
- The user requests a bounded pilot benchmark with a baseline pass versus a with-wiki pass.
- The user wants a continue/pause/redesign decision for the LLM-Wiki workflow or a specific domain.
- The user wants to check grounding, citation coverage or unsupported-claim rate on recent answers.
- The user wants an operational-health check (staleness, review backlog, freshness lag) as part of a periodic review.
- Route pure tooling/CI/framework selection questions to
llm-wiki-eval-toolinginstead.
Inputs
- LLM-Wiki vault path.
- Recent questions, queries or tasks if available.
wiki/index.md,wiki/log.md,wiki/queries/, lint reports and eval reports.- Optional evaluation question set or qrels.
- Optional adoption claim to test, such as "this wiki saves time" or "this beats RAG for our questions".
- Optional risk tier: low, medium, high, critical.
Procedure
1. Define evaluation scope
Choose one scope:
- whole vault;
- one domain;
- one project;
- recent 30/60/90-day activity;
- before/after migration;
- pilot with 20-50 sources and 10-20 realistic questions;
- high-stakes slice such as policies, runbooks or customer data procedures.
2. State the evidence level
Classify the evaluation as:
| Evidence level | Meaning |
|---|---|
| external direct | Published LLM-Wiki benchmark or implementation evidence. |
| external adjacent | GraphRAG, memory, context-engineering or RAG benchmark evidence. |
| local operational | This user's own metrics, query tests, traces and human review. |
What ships with it
3 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 11d ago First seen · 293 lines · 64 tokens per session scan A 24927a78b7c0
llm-wiki-eval is a skill published in the GitHub repository po4yka/llm-wiki-skills (3 stars, last pushed 18d ago), licensed MIT. It adds 64 tokens to every session and 2,198 once invoked, about $0.0003 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
test-agent
Systematisches Testen von BACH und anderen LLM-OS-Systemen. Orchestrator fuer Test- und Vergleichs-Workflows.
read-search-smart
Semantic search via Smart Connections embeddings. Returns ranked chunks with cosine scores and breadcrumbs. Use when the query is conceptual (meaning, not literal substring). EN triggers: "find notes about X", "what do I have on X", "semantic search for X", "find concepts related to X", "notes similar to ". FR…
knowledge-base-interop
Two-layer knowledge architecture — curate in LLM Wiki, search in Astra KB. Covers input classification, source-to-wiki ingestion, wiki-to-KB export (batch + realtime), semantic chunking, SAG extraction, health checks, and multi-format export.
lint
A health check for the LLM Wiki, a knowledge base about large language models. It looks for missing pages, broken links, conflicting or outdated content, missing fields, incomplete topic maps, weak bias checks, and an old core-context snapshot.
knowject-rag-eval
A skill for creating or reviewing evaluation cases for RAG, a system that searches stored documents before generating an answer, including the quality of its citations.
cli-eval
Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.