Getting it into your agent
It runs from inside its repository, so the clone comes first — what it calls does not travel with the file alone.
git clone --depth 1 https://github.com/heidihelena/citevahtinpx agentmods add skills/heidihelena/citevahti/citevahti-evalWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/heidihelena/citevahti/citevahti-eval)<a href="https://agentmods.dev/skills/heidihelena/citevahti/citevahti-eval"><img src="https://agentmods.dev/badge/skills/heidihelena/citevahti/citevahti-eval/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/heidihelena/citevahti/citevahti-eval"><img src="https://agentmods.dev/badge/skills/heidihelena/citevahti/citevahti-eval.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00097 | $0.01585 |
| Opus 5 | $0.00048 | $0.00792 |
| Sonnet 5 | $0.00019 | $0.00317 |
| Haiku 4.5 | $0.00010 | $0.00159 |
Grade A, and why
citevahti-eval scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 11d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 109 lines — stays where its author put it; the contents beside it link to each section on GitHub.
CiteVahti eval — measuring the tool honestly (ADR-0009)
CiteVahti's pitch is "evidence tools held to evidence standards." This skill is where that stops being copy. It is not a single human-gold release gate — that framing was wrong. Per ADR-0009, citation checking is a defence-in-depth stack (human + lexical detector + AI models), and evaluation has three separate objects, each measured differently.
The governing idea is the cheese-hole principle: safety comes from layers whose holes don't line up. So a model that merely agrees with the human adds no defence — the best layer catches what the others miss. Measurement is built around that, not around conformity.
Triggers
Use when the maintainer asks to: run or extend the automatic claim-lexicon eval; check it against baseline; re-freeze the baseline after an intended change; reason about or publish model ratings; prepare an eval-results page.
Do NOT use for: checking a researcher's manuscript (citevahti-dev), sweeping a
reference list (citevahti-screen), choosing/operating models (citevahti-models), or
the offline pytest suite (that's secure-release's build gate).
The three evaluation objects (keep them separate)
1. Claim-lexicon eval — automatic, you run it
The primary, always-on evaluation of the deterministic lexical floor (text.py). No
human-rater dependency — that is what makes it automatic.
python validation/claimcheck/eval_lexicon.py # score + per-phenomenon report
python validation/claimcheck/eval_lexicon.py --check # CI gate: exit 1 on regression
python validation/claimcheck/eval_lexicon.py --write-baseline # re-freeze after an intended change
- Ground truth is the author-labelled
expectedrelation inlexicon_cases.jsonl. - The lexical layer is one transparent slice with known holes (paraphrase/synonymy; antonym contradictions with no negation cue). The eval names the holes per phenomenon — it does not pretend they're gone; the AI-model and human layers cover them.
- Regression policy:
validation/claimcheck/acceptance-thresholds.md. Precision is floored (a flag must be worth interrupting for); recall is published, not chased — the inverted-U: over-flagging is worse than under-flagging, and widening a lexicon to chase recall is the wrong layer's job. A negated contradiction served as support must stay 0 (tests/test_claimcheck_polarity.py,tests/test_lexicon_eval.py). - Regression here blocks a release; the known-hole categories are reported, not gated.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 11d ago First seen · 109 lines · 97 tokens per session scan A 7ae5bbea8489
citevahti-eval is a skill published in the GitHub repository heidihelena/citevahti (1 stars, last pushed 5d ago), licensed Apache-2.0. It adds 97 tokens to every session and 1,585 once invoked, about $0.0005 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
ml-paper-writing
Write publication-ready ML/AI papers for NeurIPS, ICML, ICLR, ACL, AAAI, COLM. Use when drafting papers from research repos, structuring arguments, verifying citations, or preparing camera-ready submissions. Includes LaTeX templates, reviewer guidelines, and citation verification workflows.
vscode-visual-regression
Write Storybook stories and visual regression tests for the Kilo VS Code extension webview UI.
simulation-study
Scaffold and run a reproducible Monte Carlo simulation study in R — a declared assumption regime, a parameterized DGP, an estimator grid, a seeded replication loop, and a summary of bias, RMSE, empirical SE, coverage, size/power with Monte Carlo standard errors. Use when the user says "run a Monte Carlo simulation"…
vaccinate
Qualify a check before it is allowed to clear anything — prove it can detect the failure it is meant to catch. Seeds known defects into a copy of a real artifact plus a clean control, runs the checker, and reports recall and false-positive rate into a qualification ledger. Use when the user says "does this check…
r-package-check
Run the full R package release gate — regenerate docs, run the test suite, run R CMD check --as-cran, and triage every ERROR / WARNING / NOTE against CRAN policy before a release or submission. Use when the user says "check my R package", "R CMD check", "is this package CRAN-ready", "run devtools::check", "prepare for…
qa-quarto
Adversarial Quarto-vs-Beamer parity QA. A critic agent compares the Quarto HTML render to the Beamer PDF benchmark for content/visual parity; a fixer agent applies fixes; loops until APPROVED (max 5 rounds). Use when user says "qa the quarto", "check parity", "does the html match the pdf?", "quarto matches beamer?"…