Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add alivirgo/Major-AI-Skills --skill ai-evaluation-datasetgit clone --depth 1 https://github.com/alivirgo/Major-AI-SkillsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/alivirgo/major-ai-skills/ai-evaluation-dataset)<a href="https://agentmods.dev/skills/alivirgo/major-ai-skills/ai-evaluation-dataset"><img src="https://agentmods.dev/badge/skills/alivirgo/major-ai-skills/ai-evaluation-dataset/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/alivirgo/major-ai-skills/ai-evaluation-dataset"><img src="https://agentmods.dev/badge/skills/alivirgo/major-ai-skills/ai-evaluation-dataset.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00031 | $0.00330 |
| Opus 5 | $0.00015 | $0.00165 |
| Sonnet 5 | $0.00006 | $0.00066 |
| Haiku 4.5 | $0.00003 | $0.00033 |
Grade A, and why
ai-evaluation-dataset scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured today.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
What it actually says
Evaluation Dataset
Scope
Ask for the target task, deployment population, known failures, and which errors are unacceptable. Reuse the project's evaluator and data format when present; otherwise propose JSONL records with id, input, expected_behavior, forbidden_behavior, rubric, source, and split.
Procedure
Separate training examples, prompt-development examples, and held-out evaluation cases. Split by originating document, user, or conversation rather than individual rows when rows share information. Keep near-duplicates in the same split. Remove secrets and obtain permission before including private user content.
Checks
Include ordinary cases, boundary cases, ambiguous inputs, and explicit abstention cases. Label expected behavior before viewing candidate model outputs. For subjective outputs, use observable rubric criteria instead of a single preferred phrasing.
Failure Handling
Report case counts by split and slice, provenance, duplicate findings, and unresolved label disagreements. Freeze a dataset version and content hash before comparing models. Never report evaluation accuracy without actually running the evaluator.
Deliverable
Given several paraphrases of one support ticket, keep them in a single split; a random row split would leak the answer.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- today First seen · 35 lines · 31 tokens per session scan A e5d4ea7c7ac8
ai-evaluation-dataset is a skill published in the GitHub repository alivirgo/Major-AI-Skills (1 stars, last pushed today), licensed MIT. It adds 31 tokens to every session and 330 once invoked, about $0.0002 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-12.
Other skills, from other repositories
eval-harness
Professional Eval Harness Expert skill. Integrate LLM API workflows, safe system prompt guidelines, and agentic workflows.
advanced-evaluation
This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.
agent-harness-fault-injection
Use when an agent workflow needs deterministic recovery evidence for sandbox, MCP/tool, worker, checkpoint, memory, or orchestration failures.
ax-extract-workflow
Reconstruct workflow behind a past coding-agent artifact using local ax sessions/commits/skills/tool traces. Use when asked how X was built.
agenttrace-session-audit
Audit local AI coding-agent sessions with agenttrace for cost, tool failures, latency, anomalies, health, diffs, and CI gates.
ai-engineering-toolkit
6 production-ready AI engineering workflows: prompt evaluation (8-dimension scoring), context budget planning, RAG pipeline design, agent security audit (65-point checklist), eval harness building, and product sense coaching.