Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add guanyang/open-agent-hub --skill advanced-evaluationgit clone --depth 1 https://github.com/guanyang/open-agent-hubWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/guanyang/open-agent-hub/advanced-evaluation)<a href="https://agentmods.dev/skills/guanyang/open-agent-hub/advanced-evaluation"><img src="https://agentmods.dev/badge/skills/guanyang/open-agent-hub/advanced-evaluation.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00044 | $0.03543 |
| Opus 5 | $0.00022 | $0.01772 |
| Sonnet 5 | $0.00009 | $0.00709 |
| Haiku 4.5 | $0.00004 | $0.00354 |
Grade A, and why
advanced-evaluation scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 8d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
This is a copy
100% identical to advanced-evaluation — 0 lines differ, which has more behind it and is treated as the original. This page carries a canonical link to it rather than competing with it.
How it starts
The opening of the file, as written. The whole thing — 410 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Advanced Evaluation
This skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It synthesizes research from academic papers, industry practices, and practical implementation experience into actionable patterns for building reliable evaluation systems.
Key insight: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops.
When to Activate
Activate this skill when:
- Building LLM-as-judge systems for LLM outputs
- Comparing multiple model responses to select the best one
- Establishing consistent quality standards across evaluation teams
- Debugging evaluation systems that show inconsistent results
- Designing A/B tests for prompt or model changes
- Creating rubrics specifically for LLM or human/LLM hybrid judges
- Analyzing correlation between automated and human judgments
Do not activate this skill for adjacent work owned by other skills:
- General deterministic checks, regression suites, production quality gates, or outcome metrics:
evaluation. - Autonomous loop governance, locked rubrics, rollback, or PR approval boundaries:
harness-engineering. - Tool API contracts for evaluation tools:
tool-design.
Core Concepts
The Evaluation Taxonomy
Select between two primary approaches based on whether ground truth exists:
Direct Scoring — Use when objective criteria exist (factual accuracy, instruction following, toxicity). A single LLM rates one response on a defined scale. Achieves moderate-to-high reliability for well-defined criteria. Watch for score calibration drift and inconsistent scale interpretation.
Pairwise Comparison — Use for subjective preferences (tone, style, persuasiveness). An LLM compares two responses and selects the better one. Pairwise methods often correlate better with human preference than open-ended direct scoring for subjective tasks (claim-advanced-evaluation-position-swap). Watch for position bias and length bias.
What ships with it
5 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 8d ago First seen · 410 lines · 44 tokens per session scan A 8b9e3d9de07a
advanced-evaluation is a skill published in the GitHub repository guanyang/open-agent-hub (959 stars, last pushed 2d ago), licensed MIT. It adds 44 tokens to every session and 3,543 once invoked, about $0.0002 per session on Opus 5. A static security scan graded it A with 0 findings. It is 100% identical to advanced-evaluation, differing in 0 lines, and is treated as a copy.
Other skills, from other repositories
ml-data-leakage-guard
Detects and prevents data leakage in machine learning and mathematical modeling. Use after ML tasks involving data cleaning, feature engineering, data augmentation, algorithm development, normalization, missing value imputation, dimensionality reduction, feature selection, or time series modeling. Checks if…
coverage-tracker
Run a Google Alerts-style keyword coverage tracker. Uses news-search for recent keyword queries, lets the LLM dedupe and classify real features versus junk, stores decisions in SQLite, and alerts only on new real coverage.
prompt-set-qa
Gate a prompt universe for schema and provenance completeness, target or campaign contamination, evidence entailment, naturalness, one-concept clarity, architecture consistency, aided status, answer leakage, and semantic duplicates. Use after realistic prompt generation and before human panel selection.
realistic-prompt-generation
Generate natural, controlled prompt variants from a target-blind design brief and prompt architecture while preserving approved jobs, acts, journeys, constraints, roles, locales, proximity bands, and evidence language. Use after architecture design and before contamination or semantic QA.
ac-criteria-validator
Validate acceptance criteria and feature completion. Use when checking if features pass, validating test results, verifying acceptance criteria, or determining feature completion status.
ai-visibility-panel-design
Select QA-approved canonical intent cells into a versioned AI-visibility tracking panel with partitions, variants, lanes, surfaces, locales, repetitions, separate exposure and priority weights, randomization, uncertainty, refresh rules, and campaign controls. Use after prompt QA or when revising an existing panel.