Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add sigistry/marketplace --skill eval-authoringgit clone --depth 1 https://github.com/sigistry/marketplaceWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/sigistry/marketplace/eval-authoring)<a href="https://agentmods.dev/skills/sigistry/marketplace/eval-authoring"><img src="https://agentmods.dev/badge/skills/sigistry/marketplace/eval-authoring/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/sigistry/marketplace/eval-authoring"><img src="https://agentmods.dev/badge/skills/sigistry/marketplace/eval-authoring.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00096 | $0.00957 |
| Opus 5 | $0.00048 | $0.00478 |
| Sonnet 5 | $0.00019 | $0.00191 |
| Haiku 4.5 | $0.00010 | $0.00096 |
Grade A, and why
eval-authoring scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 6d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 40 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Eval Authoring
Purpose
Give LLM apps a repeatable way to escape "vibes-based" development, where a prompt is changed, a few outputs are eyeballed, and the change ships with no evidence it helped. An eval suite turns that into a measured, versioned, CI-gated check: every prompt or model change is scored against fixed cases, and a regression fails the build. This skill standardizes what to test, how to grade it, and how to wire it in, riding an existing framework (promptfoo, DeepEval, Ragas) when one is present, and falling back to plain unit tests plus an LLM judge when none is.
The three grader types
| Grader | Use for | Cost / stability | Example |
|---|---|---|---|
| Assertion | Deterministic, checkable facts about the output | Cheap, stable, run every commit | is-json, json-schema, contains, regex, not-contains (banned strings), latency/cost budget |
| LLM-as-judge | Qualities assertions can't capture | Costs a call, needs a threshold | Faithfulness, helpfulness, tone, "answers the question", pairwise vs. baseline |
| Golden dataset | Regression over a curated set of real cases | Grows from production traffic | input → expected/assert, tagged, versioned in the repo |
Prefer assertions wherever the property is checkable, they are free and non-flaky. Reach for a judge only for genuinely subjective qualities, and always give it a pass threshold. Use the golden set to lock in behavior you've already fixed so it never regresses.
Authoring workflow
- Name the contract. For the prompt/agent under test: its inputs, the output shape it promises, and the failure modes worth catching (wrong format, missing field, hallucination, ignored instruction, unsafe content, verbosity).
- Collect real cases. Pull 3–6 representative inputs from fixtures, logs, or the prompt's own examples. Never invent domain facts, a wrong "expected" answer is worse than no test.
- Layer the graders. Start with assertions for everything checkable; add judge tests for the rest; seed a golden file with the cases.
- Set thresholds. Every judge assertion needs a numeric pass bar. For regression gating, prefer pairwise (new output must be ≥ baseline) over an absolute score.
- Gate CI. Run the suite on PRs and fail on any assertion failure or judge score below threshold/baseline. Keep model-calling tests out of the every-commit path where possible.
- Grow the set. Every production failure becomes a new golden case, so the same bug can never ship twice.
What ships with it
2 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 6d ago First seen · 40 lines · 96 tokens per session scan A 9c2cff8ed358
eval-authoring is a skill published in the GitHub repository sigistry/marketplace (3 stars, last pushed 5d ago), licensed MIT. It adds 96 tokens to every session and 957 once invoked, about $0.0005 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-03.
Other skills, from other repositories
deep-plan
Creates detailed, sectionized, TDD-oriented implementation plans through research, stakeholder interviews, and multi-LLM review. Use when planning features that need thorough pre-implementation analysis.
tdd
Test-driven development — the red-green-refactor discipline for code that's correct by design, not by accident. INVOKE PROACTIVELY whenever writing or changing code, fixing a bug, adding or structuring tests, or choosing what to mock — even when nobody says "TDD" or "tests". (Tests are code held to…
wave-execution-framework-v2
Use when executing multi-wave engineering work needing strict TDD, bug-capture/fix split, quality gates, and orchestrated teams with per-agent Opus-advisor / Sonnet-executor model routing.
nano-banana
This skill should be used for Python scripting and Gemini image generation. Use when users ask to generate images, create AI art, edit images with AI, or run Python scripts with uv. Trigger phrases include "generate an image", "create a picture", "draw", "make an image of", "nano banana", or any image generation…
dotnet-verify
This skill should be used when working with Verify snapshot tests in .NET projects. Use when updating verified snapshots after intentional code changes, accepting new snapshots, discovering verify tests, or troubleshooting snapshot mismatches. Trigger phrases include "verify tests", "update snapshots", "accept…
auto-loop
TDD-based autonomous development loop with checkpoint recovery and observability changelog.