What is generate-evals?

A tool for creating and checking evaluation scenarios for a repository plugin. Each scenario describes a task and the criteria used to judge whether the plugin changes an agent's behaviour, and can be checked offline.

Use it to inventory a plugin, create scenarios in the tessl-compatible evals format, identify coverage gaps, and verify the scenarios locally.

It provides repeatable checks for skills, commands, and hooks instead of relying only on manual inspection.

generate-evals — Skill for Claude Code by rikdc/ai-skills

Install

Getting it into your agent

It runs from inside its repository, so the clone comes first — what it calls does not travel with the file alone.

Clone the repo

git clone --depth 1 https://github.com/rikdc/ai-skills

agentmods

npx agentmods add skills/rikdc/ai-skills/generate-evals

Made for: Claude Code.

Or install eval-workflow, the plugin that ships this one along with the rest of its 2 skills.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/skills/rikdc/ai-skills/generate-evals/github.svg)](https://agentmods.dev/skills/rikdc/ai-skills/generate-evals)

Your own site

<a href="https://agentmods.dev/skills/rikdc/ai-skills/generate-evals"><img src="https://agentmods.dev/badge/skills/rikdc/ai-skills/generate-evals/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/skills/rikdc/ai-skills/generate-evals"><img src="https://agentmods.dev/badge/skills/rikdc/ai-skills/generate-evals.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 108 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 3,042 The whole file, excluding the scripts and references it only reads on demand.

Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00108	$0.03042
Opus 5.5	$0.00043	$0.01217
Sonnet 5.5	$0.00022	$0.00608
Haiku 4.5	$0.00011	$0.00304

Measured 3d ago against content hash 2a9f98324d86, method: parsed. Prices are Anthropic first-party input rates as of 2026-10-07, from the pricing page.

Security

Grade A, and why

generate-evals scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 3d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

plugins/eval-workflow/skills/generate-evals/SKILL.md · 213 lines

How it starts

The opening of the file, as written. The whole thing — 213 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Generate Evals - eval scenarios for repository plugins

Builds and verifies eval scenarios that prove a plugin's skills, commands, or hooks change agent behaviour. Scenarios live inside the plugin directory at evals/, so tessl eval run plugins/<name> can run them against the plugin. plugins/git-workflow/evals/ in this repository is the canonical worked example - read one of its scenarios before writing new ones.

Inputs

<plugin-path>: the plugin directory relative to the repository root (for example plugins/dev-skills).
--count N: target scenarios per unit. Default is 3; fewer is acceptable for small units. One "unit" is a skill, a command, or a hook script.
--dry-run: print the scenario list and any coverage gaps; write nothing.

Workflow

1. Inventory the plugin

Read .claude-plugin/plugin.json, the README, and the directory tree.
Classify the content: skills/<name>/SKILL.md, commands/*.md, hooks/*.sh (or scripts exercised as hooks), or a mix.
Note existing evals/ content so new scenario names do not collide with it.
For hooks, also read the relevant tests/test-*.sh to understand the contract - but never let a scenario duplicate unit-level test coverage.

2. Standing structure, manifests and boundaries

Scenarios live at the plugin root: plugins/<plugin>/evals/<scenario>/, beside skills/ - never inside a skill directory and never at the repository root.
A plugin must have its tessl manifest before its evals can run. If plugins/<plugin>/.tessl-plugin/plugin.json is missing, create it from the Claude manifest: "name" is the workspace (the first segment of the root tessl.json name) plus / and the plugin name, "version" and "description" are copied from .claude-plugin/plugin.json, and "private": true. Add the pointers the plugin ships - "skills": "./skills/", "commands": "./commands/". Note that tessl plugin lint rejects a manifest with none of skills, commands, rules, hooks or nativeHooks, so a hooks-only plugin must declare its hooks in the tessl hooks schema (event name, matcher, and a command pointing at the bundled script through ${TESSL_PLUGIN_DIR}). If the Claude manifest is missing or malformed, stop and ask.
HARD BOUNDARY: never modify .claude-plugin/**, hook scripts, hook registration, or a skill body. Scenarios add files, plus a manifest when one is absent.

Read the full file on GitHub · 213 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

3d ago First seen · 213 lines · 108 tokens per session scan A 2a9f98324d86

Subscribe to this mod's changes

generate-evals is a skill published in the GitHub repository rikdc/ai-skills (2 stars, last pushed 2d ago), licensed MPL-2.0. It adds 108 tokens to every session and 3,042 once invoked, about $0.0004 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-10-16.

Building agents Claude Code Testing claude-code-plugin

live-preview

Mid-build visual verification loop. Takes screenshots of components during construction, not just after. Catches visual regressions and invisible features before they compound. Requires Playwright or similar screenshot tool.

SethGammon/Citadel · 40 tokens

brooks-sweep

Full-sweep mode: runs a unified analysis across all quality dimensions — code decay, architecture, tech debt, and test quality — then applies fixes directly to the codebase. Safe changes are auto-applied; risky changes are confirmed before execution. Drawing on twelve classic engineering books. Triggers when: user…

hyhmrright/brooks-lint · 178 tokens

generate-evals