Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add s977043/river-review --skill eval-driven-skill-designgit clone --depth 1 https://github.com/s977043/river-reviewWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/s977043/river-review/eval-driven-skill-design)<a href="https://agentmods.dev/skills/s977043/river-review/eval-driven-skill-design"><img src="https://agentmods.dev/badge/skills/s977043/river-review/eval-driven-skill-design/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/s977043/river-review/eval-driven-skill-design"><img src="https://agentmods.dev/badge/skills/s977043/river-review/eval-driven-skill-design.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00051 | $0.02775 |
| Opus 5.5 | $0.00020 | $0.01110 |
| Sonnet 5.5 | $0.00010 | $0.00555 |
| Haiku 4.5 | $0.00005 | $0.00278 |
Grade A, and why
Eval-Driven Skill Design scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 11d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 179 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Pattern declaration
Primary pattern: Reviewer Secondary patterns: Tool Wrapper Why: skill 追加・重要変更時に happy-path / guard fixture、eval 配線、paired ablation の評価可能性を診断し、存在・発火と実効性を混同しないための educator スキル。fixture-only / docs-only PR では起動しない。
Goal / 目的
- 新規
skills/**/SKILL.mdを追加する PR が 検出ケース (happy-path) と抑制ケース (guard) のペア を fixture として持つことを確認する。 eval/promptfoo.yaml相当の評価設定またはtests/fixtures/repo-wide-eval/への登録が用意されているかを確認する。- 欠けている場合は
npm run eval:fixtures/npm run eval:repo-contextをどう通すかを案内する。 - 新規・重要変更された skill について、同じ case / runtime / model / effort で WITH skill と WITHOUT skill（または旧版 baseline と candidate） を比較できるか確認する。
installed/selected/firedを effectiveness の証拠にせず、寄与が不明ならINCONCLUSIVEとして追加 Evidence を要求する。
Guidance
Eval cycle 全体像 (#688 で整備済)
new SKILL.md
├─ fixtures/
│ ├─ <NN>-happy.md # 検出されるべき diff（必須 1 件以上）
│ └─ <NN>-guard.md # 検出してはいけない diff（必須 1 件以上）
├─ golden/<NN>-*.md # promptfoo similar 評価用の代表出力
├─ prompt/{system,user}.md # promptfoo prompts:
└─ eval/promptfoo.yaml # contains / not-contains / llm-rubric / similar
または tests/fixtures/repo-wide-eval/ に同じ skill の {happy, guard} ケースを追加し、tests/fixtures/repo-wide-eval/cases.json に登録する形でも eval lift が測定できる。
Marginal contribution / paired ablation
fixture / golden output は「壊れていないか」を見る regression evidence です。skill が存在する価値そのものは、可能な範囲で baseline と candidate の差分として測ります。
same case / runtime / model / effort
├─ WITHOUT skill (or previous-version baseline)
└─ WITH skill (or candidate)
↓
compare delta
原則:
- Usage != Effectiveness:
installed/selected/firedは availability / activation evidence であり、品質改善の証拠ではない。 - 比較では skill 以外の条件を固定する。model / runtime / effort / input が違う結果を skill の寄与として扱わない。
- 非決定的な実行は単発結果で断定せず、複数 trial と paired case を使う。
- 最低限、detection / false positive / critical regression を比較し、取得可能なら time / token / cost / human intervention も見る。
- candidate 側が実際に発火していない、paired case が不足している、sample が足りない場合は
INCONCLUSIVE。自動的に「効果なし」へ倒さない。 - River Review 内の Harness 改善では、独自比較器を増やさず既存の Experiment Manifest / paired replay（
src/lib/paired-replay.mjs、docs/development/1574-p2-paired-replay.md）を優先して再利用する。 - model / runtime の major update、skill の責務・prompt・routing の大幅変更時は再評価する。旧モデルで有効だった skill が新モデルでは冗長になる可能性を前提にする。
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 11d ago Changed · +35 lines · -12 tokens per session 3b7b2f9197c0
- 1mo ago First seen · 144 lines · 63 tokens per session scan A 46b51ba04399
Eval-Driven Skill Design is a skill published in the GitHub repository s977043/river-review (7 stars, last pushed today), licensed MIT. It adds 51 tokens to every session and 2,775 once invoked, about $0.0002 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-03.
Other skills, from other repositories
testing-strategy
Design test strategies and test plans with coverage targets. Complements /draft:coverage which measures what this skill plans. Auto-loaded by /draft:implement before TDD.
plan
Analyzes architecture, selects patterns, assesses testability, then decomposes work into ordered TDD tasks with exact verification commands and explicit acceptance mapping. Works from an approved spec (zuvo:brainstorm output) or directly from a user-provided description.
run-iteration-eval
Run the Logic-Lens content-eval pipeline for one iteration and produce a scored summary.json — use to measure a skill change. Wraps scripts/run-content-evals.sh (runner, costs tokens) and scripts/grade-iteration.py (grader, free, re-runnable). ALWAYS sync the plugin cache first. Use when the user wants to "run the…
x-qdev
A compact development workflow for a small, clearly defined code change. It keeps the requirement, initially failing tests, implementation, and real verification results in one task document, following TDD, or test-driven development.
test-audit
Audit test suites for T1-T4 violations using AST analysis, mock detection, and multi-stage synthesis. Invoke when user asks to audit tests, check test quality, find mock violations, review test effectiveness, or inspect test suites for over-mocking. Triggers automatic rewrites when quality gates fail.
validate-plan
Validate implementation plans against DRY, YAGNI, TDD principles and best practices. Use when reviewing plans from pocket-planning or another planning workflow to identify gaps, anti-patterns, and improvement opportunities before execution.