Getting it into your agent
It runs from inside its repository, so the clone comes first — what it calls does not travel with the file alone.
git clone --depth 1 https://github.com/iliaal/whetstonenpx agentmods add commands/iliaal/whetstone/eval-skillsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/commands/iliaal/whetstone/eval-skills)<a href="https://agentmods.dev/commands/iliaal/whetstone/eval-skills"><img src="https://agentmods.dev/badge/commands/iliaal/whetstone/eval-skills.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00022 | $0.01926 |
| Opus 5 | $0.00011 | $0.00963 |
| Sonnet 5 | $0.00004 | $0.00385 |
| Haiku 4.5 | $0.00002 | $0.00193 |
Grade A, and why
eval-skills scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 7d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 121 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Evaluate and rank all skills
Score every skill that has sufficient harvested eval data, rank them by procedure-following score, and identify the best candidates for /evolve-skill.
Arguments
MIN_EXAMPLES=30 (minimum harvested examples to include a skill, default: 30)
TOP=10 (how many bottom-ranked skills to highlight, default: 10)
Parse from: $ARGUMENTS
Pipeline
Step 1: Harvest fresh data
python3 distillery/scripts/distiller.py harvest-sessions
Capture the JSON output. Extract the skills dict to know which skills have data and how many examples each has.
Step 2: Identify eligible skills
From the harvest output, list skills with count >= MIN_EXAMPLES. Exclude _unattributed. Sort by example count descending.
Present a table (include the ambiguous count — it is the dominant class post-2026-07-07 and the split is meaningless without it):
| Skill | Examples | Positive | Negative | Ambiguous |
|--------------------------------|----------|----------|----------|-----------|
| ia-code-review | 438 | 0 | 3 | 435 |
| ... | | | | |
Read the columns honestly:
- ambiguous — no typed user outcome. This is the NORMAL case for subagent sessions (they end without a human reply), so a high ambiguous count is expected, not a problem.
- positive — requires 2+ typed user messages with satisfaction signal; rare for subagent-driven skills.
- negative — a genuine typed user correction. Low counts (0-3) are the norm now; each one is high-signal.
Step 3: Eval each eligible skill (in-session sub-agents)
The judging runs as in-session sub-agents (no billed claude -p). For each eligible skill, build a golden set then emit judge tasks.
RECOMMENDED (human-label) build, since harvest data is mostly ambiguous and approve-golden hard-errors on ungraded labels:
python3 distillery/scripts/distiller.py build-golden <skill> --top 20
# → edit candidates.jsonl labels to positive / negative / skip, then:
python3 distillery/scripts/distiller.py approve-golden <skill>
python3 distillery/scripts/distiller.py dspy-eval <skill> --dataset golden --max-examples 10 --emit-tasks
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 7d ago First seen · 121 lines · 22 tokens per session scan A ffa0c8913ebb
eval-skills is a command published in the GitHub repository iliaal/whetstone (33 stars, last pushed 8d ago), licensed MIT. It adds 22 tokens to every session and 1,926 once invoked, about $0.0001 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other commands, from other repositories
composite-actions
Generate, review, secure, and test composite GitHub Actions following best practices — full repo scaffold, interview-driven generation, PR creation on existing repos, SHA pinning, secrets-as-inputs, job summaries, and actionlint validation.
github-actions
Design, review, secure, and debug GitHub Actions workflows — reusable workflows, OIDC federation, SHA pinning, token scoping, promotion orchestration, and CI failure diagnosis.
datadog
Set up and troubleshoot Datadog — Agent deployment on Kubernetes, APM instrumentation, Log Management, Monitors, Dashboards, SLOs, Synthetic tests, and live incident investigation using the Datadog MCP server. Covers Terraform-managed Datadog resources.
fluxcd
FluxCD entry point — routes to the right workflow based on what you need. Live cluster issue → structured 5-workflow debug trace. Repo health check → 6-phase audit (discovery, validation, API compliance, best practices, security). Helm chart review → helmchart. Starts by asking one question to confirm the right mode.
linkerd
Linkerd-specific diagnostics — mTLS verification, proxy injection issues, authorization policy debugging, traffic management, and multi-cluster connectivity problems.
terraform
Runs through the full Terraform validation pipeline — fmt, validate, tflint, security scan — and reviews a module or plan for blast radius, IAM risk, and state impact.