Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add malloydata/publisher --skill eval-judgegit clone --depth 1 https://github.com/malloydata/publisherWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/malloydata/publisher/eval-judge)<a href="https://agentmods.dev/skills/malloydata/publisher/eval-judge"><img src="https://agentmods.dev/badge/skills/malloydata/publisher/eval-judge.svg" alt="Measured on agentmods" height="20"></a>- NVIDIA SkillSpector warn
SkillSpector: 1 finding, up to medium
These are SkillSpector’s own severities. On a checked sample its high-severity flags on skills were ~96% false positives — a documented command, a public API, a “never do X” rule — so we show them as a caution to read, not a verdict. Why →
- medium Agent Snooping · line 26 Skill enumerates or reads other installed skills. Access to other skills' SKILL.md files or the skills directory reveals prompt instructions, capabilities, and secrets that should be invisible to peer skills.Fix: Remove all code or instructions that list or read other skills' files or directories. Skills should operate independently; cross-skill access is a privilege escalation.
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00083 | $0.02306 |
| Opus 5 | $0.00042 | $0.01153 |
| Sonnet 5 | $0.00017 | $0.00461 |
| Haiku 4.5 | $0.00008 | $0.00231 |
Grade A, and why
eval-judge scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 4d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 180 lines — stays where its author put it; the contents beside it link to each section on GitHub.
The judge
JUDGE_VERSION: 4
This skill IS the judge. One fresh judge subagent is spawned per attempt, with this skill installed in its workspace and the case materials in its prompt. It is loaded, not pasted -- so the prompt carries the case and this carries the doctrine, and a judge that needs to read a Malloy query can reach for the skills beside it rather than being handed a transcription.
Measured when it stopped being pasted, on the case that had oscillated (a valued golden against a model with no trace of the concept):
pasted into the prompt match / no_match / match / match
loaded as this skill no_match x4, and the reasoning cites the rule
It costs about 2.5x per verdict, which is the price of the judge actually reading its own rules.
Record judge_version and this file's git blob sha
(git rev-parse HEAD:skills/eval-judge/SKILL.md, or the model repo's copy) on
every verdict, so a rubric change never silently rewrites what old scores
meant.
The judge is not blind. It sees the golden. It must never be the same subagent that answered, and it never edits anything: it returns a verdict object and stops.
Read one of these before you decide
This file is the decision procedure. Four situations have their own rules, and each is a file beside this one. Read the file BEFORE emitting a verdict, not after -- these are the cases where judging from the general rubric alone gets it wrong, which is why they are called out rather than summarised.
| If | Read |
|---|---|
| the answer declines, or gives no value at all | reference/refusal.md |
| the golden itself looks wrong to you | reference/suspect-goldens.md |
| you are judging retrieval, not an answer | reference/retrieval-judge.md |
| you are AUTHORING a case rather than judging one | reference/writing-rubrics.md |
The first row is the one that catches people. A refusal is only exempt from
containment when golden.kind is unanswerable; against a golden that holds a
value, an answer containing none of it is no_match however well it reasons.
reference/refusal.md is the whole rule.
What ships with it
4 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 4d ago First seen · 180 lines · 83 tokens per session scan A 8e698227e286
eval-judge is a skill published in the GitHub repository malloydata/publisher (100 stars, last pushed yesterday), licensed MIT. It adds 83 tokens to every session and 2,306 once invoked, about $0.0004 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-03.
Other skills, from other repositories
dbt-expert
Expert-level dbt (data build tool), models, tests, documentation, incremental models, macros, and Jinja templating. Use when the user mentions analytics engineering, SQL, data transformation, Jinja, or testing, or when the task involves Project Structure and Configuration, Sources and Staging Models, Intermediate and…
reimagine-it-audit
Design Health — runs 19 deterministic quality checks on HTML output. Use when the user says /reimagine-it audit, "audit this page", "check design quality", or wants to verify craft-floor compliance before shipping. Catches blank plates, missing focus rings, non-compositor motion, off-palette accents, dead motion…
reimagine-it-extract
Emit the content signals reimagine-it reads from an HTML file — title, anchors, proper nouns, dates, numbers, emails, links, source hex colors, and the derived palette — as JSON without generating a redesign. Use when the user says /reimagine-it extract, "what does the engine see in this page", "extract the palette"…
reusable-visualization
Build ONE reusable chart visualization component that receives its data and its settings from the host application instead of fetching them, and declares the fields and config options the host exposes to viewers. Use this whenever a single chart component is reused across many different queries rather than built for…
lightdash-agent-slack-messaging
Use this skill when writing, designing, or generating Slack messages for Lightdash's in-app analytics agent. Triggers when someone asks to create agent update messages, Slack digests, agent notifications, weekly summaries, daily summaries, or any Slack copy for the Lightdash project agent. Also use when asked to vary…
developing-in-lightdash
Use when reading, creating, and editing Lightdash dashboards and charts as JSON, including dashboard layout and chart-type-specific configuration.