Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add shimo4228/claude-harness --skill llm-as-judgegit clone --depth 1 https://github.com/shimo4228/claude-harnessWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/shimo4228/claude-harness/llm-as-judge)<a href="https://agentmods.dev/skills/shimo4228/claude-harness/llm-as-judge"><img src="https://agentmods.dev/badge/skills/shimo4228/claude-harness/llm-as-judge/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/shimo4228/claude-harness/llm-as-judge"><img src="https://agentmods.dev/badge/skills/shimo4228/claude-harness/llm-as-judge.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00116 | $0.02322 |
| Opus 5 | $0.00058 | $0.01161 |
| Sonnet 5 | $0.00023 | $0.00464 |
| Haiku 4.5 | $0.00012 | $0.00232 |
Grade A, and why
llm-as-judge scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 12d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 197 lines — stays where its author put it; the contents beside it link to each section on GitHub.
LLM-as-Judge — Checks as Evidence, Holistic Verdict, No Scores
Core rule in one line:
Collect evidence with binary Yes/No checks, let the LLM issue one named holistic verdict, and never aggregate the answers into a score.
Why not rubric scores
- Irreproducible. Numeric scores on the same input drift between runs (3 vs 4 out of 5). LLMs are bad at independent per-dimension scoring — they get pulled by the overall impression, and central-tendency bias compresses scores into a narrow band that straddles any threshold you pick.
- Dilution. A sum or satisfaction ratio converts one fatal defect ("the referenced file does not exist") into a small deduction. One dominant No must be able to decide the outcome alone.
- Unexplainable. Nobody can say why something is a 3.5. A No answer to a concrete question explains itself and doubles as the improvement item.
Rubrics were invented to structure human evaluation, where assessors can score dimensions independently. For an LLM, invert the design: force the checks it tends to skip, and leave the judgment holistic — that is what it is good at.
The three principles
| Principle | Do | Don't |
|---|---|---|
| ① Binary checks | Decompose criteria into Yes/No questions with 1-line evidence each | "Rate specificity 1–5" |
| ② Named holistic verdict | Pick exactly one verdict from a fixed enum, judging the whole | "Total 12 points → pass" |
| ③ No aggregation | Enumerate the No answers as the verdict's rationale | Use the Yes-ratio as a quality metric |
① Binary checks
Ask "is there a runnable command example? Yes/No", not "how specific is this?". Good binary questions are verifiable (the text settles them black-or-white) and evidentiary (a No names what is missing). Require one line of evidence per answer — a file read, a path check, a measured command output.
② Named holistic verdict
Holistic judgment — one conclusion from the whole, without passing through per-dimension numbers — is the LLM's strength. Design the verdict enum so each value maps 1:1 to a next action, e.g.:
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 12d ago First seen · 197 lines · 116 tokens per session scan A a5c8edd90c02
llm-as-judge is a skill published in the GitHub repository shimo4228/claude-harness (3 stars, last pushed 6d ago), licensed MIT. It adds 116 tokens to every session and 2,322 once invoked, about $0.0006 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
ai-evaluation
Systematic evaluation (evals) for LLM and AI products. Design test cases, measure accuracy/quality, track regressions, benchmark models, and build continuous evaluation pipelines. Distinct from traditional software testing with probabilistic outputs.
evaluating-llms-harness
Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.
cli-eval
Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.
model-merging
Merge multiple fine-tuned models using mergekit to combine capabilities without retraining. Use when creating specialized models by blending domain-specific expertise (math + coding + chat), improving performance beyond single models, or experimenting rapidly with model variants. Covers SLERP, TIES-Merging, DARE, Task…
data-quality-frameworks
Implement data quality validation with Great Expectations, dbt tests, and data contracts. Use when building data quality pipelines, implementing validation rules, or establishing data contracts.
dbt-transformation-patterns
Master dbt (data build tool) for analytics engineering with model organization, testing, documentation, and incremental strategies. Use when building data transformations, creating data models, or implementing analytics engineering best practices.