What is skill-eval-scaffolder?

A scaffolding tool that creates schema-valid evaluation files for repository skills. These files describe test cases and the results a skill is expected to produce.

Use it to preview or create an evals.json manifest, add explicit and negative-control cases, and validate an existing evaluation file.

It gives a new skill a consistent starting point for behavioral coverage without running live language-model evaluations.

skill-eval-scaffolder — Skill for Claude Code by wyattowalsh/agents

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

Any agent

npx skills add wyattowalsh/agents --skill skill-eval-scaffolder

Clone the repo

git clone --depth 1 https://github.com/wyattowalsh/agents

Made for: Claude Code.

Or install agents, the plugin that ships this one along with the rest of its 71 skills, 20 agents, 5 hooks.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/skills/wyattowalsh/agents/skill-eval-scaffolder/github.svg)](https://agentmods.dev/skills/wyattowalsh/agents/skill-eval-scaffolder)

Your own site

<a href="https://agentmods.dev/skills/wyattowalsh/agents/skill-eval-scaffolder"><img src="https://agentmods.dev/badge/skills/wyattowalsh/agents/skill-eval-scaffolder/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/skills/wyattowalsh/agents/skill-eval-scaffolder"><img src="https://agentmods.dev/badge/skills/wyattowalsh/agents/skill-eval-scaffolder.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 37 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 447 The whole file, excluding the scripts and references it only reads on demand.

Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00037	$0.00447
Opus 5.5	$0.00015	$0.00179
Sonnet 5.5	$0.00007	$0.00089
Haiku 4.5	$0.00004	$0.00045

Measured 1mo ago against content hash 53ecdb4963e8, method: parsed. Prices are Anthropic first-party input rates as of 2026-10-07, from the pricing page.

Security

Grade A, and why

skill-eval-scaffolder scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.

The scan reads SKILL.md. This mod also ships 3 executable files (scripts/asset_toolkit/validate_hooks.py, scripts/check.py, scripts/scaffold_evals.py), listed below but not scanned — reading those needs a real analyzer, not pattern matching.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

skills/skill-eval-scaffolder/SKILL.md · 61 lines

What it actually says

Skill Eval Scaffolder

Create minimal, schema-valid eval manifests for repository skills.

Scope: Eval scaffolding only. Do not run live behavioral evals unless the maintainer explicitly approves.

Dispatch

$ARGUMENTS	Action
`<name>`	Preview scaffold for `skills/<name>/evals/evals.json`
`<name> --apply`	Write scaffold when file is missing or user approved overwrite
`check <name>`	Validate an existing eval manifest
Empty	Show usage gallery and manifest schema summary

Workflow

Confirm skills/<name>/SKILL.md exists.
Run the bundled scaffold script:

uv run python skills/skill-eval-scaffolder/scripts/scaffold_evals.py <name>

Add explicit, implicit, and negative-control cases before claiming adequate coverage.
Validate with the bundled check script.

Manifest Shape

Each evals/evals.json uses:

skill_name — kebab-case directory name
evals — non-empty list of cases with id, prompt, expected_output, optional files, assertions

Validation Contract

uv run python skills/skill-eval-scaffolder/scripts/check.py

Critical Rules

Never overwrite an existing eval manifest without explicit user approval.
Include at least one explicit-invocation case in new scaffolds.
Run validate_evals.py before declaring the scaffold complete.
Route live eval execution to skill-creator eval mode or maintainer CI workflows.
Keep prompts grounded in the skill dispatch table and NOT-for boundaries.

Files

What ships with it

4 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

1mo ago First seen · 61 lines · 37 tokens per session scan A 53ecdb4963e8

Subscribe to this mod's changes

skill-eval-scaffolder is a skill published in the GitHub repository wyattowalsh/agents (5 stars, last pushed 1mo ago), licensed MIT. It adds 37 tokens to every session and 447 once invoked, about $0.0001 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-03.

Building agents Testing

research-engineer

An uncompromising Academic Research Engineer. Operates with absolute scientific rigor, objective criticism, and zero flair. Focuses on theoretical correctness, formal verification, and optimal implementation across any required technology.

davila7/claude-code-templates · 43 tokens

eval-dataset-design

A guide to creating reliable test sets for AI agents. Each evaluation task includes the request, the simulated user's behaviour, a reset starting state, and a success check that can be independently verified.

bojieli/ai-agent-book · 138 tokens

tika-eval-compare

Compare extracts from two Tika builds over a corpus to detect regressions in content, encoding, exceptions, and embedded-document handling. Use for "compare before/after extracts", "eval this change against the corpus".

apache/tika · 50 tokens

cua-driver

Use Cua Driver for desktop or browser tasks that are awkward or unavailable through Bash/APIs, or when the user explicitly wants GUI interaction: app testing, visual bug reproduction, form filling, calendar entry, screenshots, and demo recording. Also covers Cua setup; not OpenAI Codex Computer Use or web research.

davidondrej/skills · 68 tokens

jetson-validate-image

Use after jetson-flash-image to run static BSP checks, on-target smoke/regression tests on a flashed DUT, or both. Not for build or flash steps. Triggers: validate bsp, on-target validation.

NVIDIA/skills · 50 tokens

atmos-validation

Validate Atmos projects, components, arbitrary JSON Schema inputs, EditorConfig, and GitHub Actions; use affected-file selection and native CI annotations.

cloudposse/atmos · 31 tokens