Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add malloydata/publisher --skill eval-judgegit clone --depth 1 https://github.com/malloydata/publisherWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/malloydata/publisher/eval-judge)<a href="https://agentmods.dev/skills/malloydata/publisher/eval-judge"><img src="https://agentmods.dev/badge/skills/malloydata/publisher/eval-judge/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/malloydata/publisher/eval-judge"><img src="https://agentmods.dev/badge/skills/malloydata/publisher/eval-judge.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- NVIDIA SkillSpector warn
SkillSpector: 1 finding, up to medium
These are SkillSpector’s own severities. On a checked sample its high-severity flags on skills were ~96% false positives — a documented command, a public API, a “never do X” rule — so we show them as a caution to read, not a verdict. Why →
- medium Agent Snooping · line 26 Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.Fix: Remove all code or instructions that list or read other skills' files or directories. Skills should operate independently; cross-skill access is a privilege escalation.
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00083 | $0.03348 |
| Opus 5.5 | $0.00033 | $0.01339 |
| Sonnet 5.5 | $0.00017 | $0.00670 |
| Haiku 4.5 | $0.00008 | $0.00335 |
Grade A, and why
eval-judge scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 15d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 246 lines — stays where its author put it; the contents beside it link to each section on GitHub.
The judge
JUDGE_VERSION: 6
This skill IS the judge. One fresh judge subagent is spawned per attempt, with this skill installed in its workspace and the case materials in its prompt. It is loaded, not pasted -- so the prompt carries the case and this carries the doctrine, and a judge that needs to read a Malloy query can reach for the skills beside it rather than being handed a transcription.
Measured when it stopped being pasted, on the case that had oscillated (a valued golden against a model with no trace of the concept):
pasted into the prompt match / no_match / match / match
loaded as this skill no_match x4, and the reasoning cites the rule
It costs about 2.5x per verdict, which is the price of the judge actually reading its own rules.
Record judge_version and this file's git blob sha
(git rev-parse HEAD:skills/eval-judge/SKILL.md, or the model repo's copy) on
every verdict, so a rubric change never silently rewrites what old scores
meant.
The judge is not blind. It sees the golden. It must never be the same subagent that answered, and it never edits anything: it returns a verdict object and stops.
Read one of these before you decide
This file is the decision procedure. Four situations have their own rules, and each is a file beside this one. Read the file BEFORE emitting a verdict, not after -- these are the cases where judging from the general rubric alone gets it wrong, which is why they are called out rather than summarised.
| If | Read |
|---|---|
| the answer declines, or gives no value at all | reference/refusal.md |
| the golden itself looks wrong to you | reference/suspect-goldens.md |
| you are judging retrieval, not an answer | reference/retrieval-judge.md |
| you are AUTHORING a case rather than judging one | reference/writing-rubrics.md |
The first row is the one that catches people. A refusal is only exempt from
containment when golden.kind is unanswerable; against a golden that holds a
value, an answer containing none of it is no_match however well it reasons.
reference/refusal.md is the whole rule.
What ships with it
4 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 15d ago Changed 26661dc2eb1c
- 1mo ago Changed · +44 lines 1ec87bd81b4d
- 1mo ago Changed · +22 lines 9c489f6ae339
- 1mo ago First seen · 180 lines · 83 tokens per session scan A 8e698227e286
eval-judge is a skill published in the GitHub repository malloydata/publisher (118 stars, last pushed today), licensed MIT. It adds 83 tokens to every session and 3,348 once invoked, about $0.0003 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-03.
Other skills, from other repositories
dbt-expert
Expert-level dbt (data build tool), models, tests, documentation, incremental models, macros, and Jinja templating. Use when the user mentions analytics engineering, SQL, data transformation, Jinja, or testing, or when the task involves Project Structure and Configuration, Sources and Staging Models, Intermediate and…
reimagine-it-extract
Emit the content signals reimagine-it reads from an HTML file — title, anchors, proper nouns, dates, numbers, emails, links, source hex colors, and the derived palette — as JSON without generating a redesign. Use when the user says /reimagine-it extract, "what does the engine see in this page", "extract the palette"…
reimagine-it-audit
Design Health — runs 19 deterministic quality checks on HTML output. Use when the user says /reimagine-it audit, "audit this page", "check design quality", or wants to verify craft-floor compliance before shipping. Catches blank plates, missing focus rings, non-compositor motion, off-palette accents, dead motion…
data-validation-contracts
Use when data crosses a boundary into your code (file, API, warehouse table, upload, model input or output), when a join or cleaning step can change row counts, or when adding pandera, Great Expectations, Pydantic or dbt checks.
reusable-visualization
Build ONE reusable chart visualization component that receives its data and its settings from the host application instead of fetching them, and declares the fields and config options the host exposes to viewers. Use this whenever a single chart component is reused across many different queries rather than built for…
sdk-features
Map Lightdash SDK capabilities to their host-UI names and the app-code wiring each needs. Use when the user asks about a feature by name (Inspect data, drill-down, exports, shareable URLs), when offering newly available features after a template upgrade, or when wiring a host-facing capability into the app.