Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add cyanheads/evals-mcp-server --skill add-toolgit clone --depth 1 https://github.com/cyanheads/evals-mcp-serverWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/cyanheads/evals-mcp-server/add-tool)<a href="https://agentmods.dev/skills/cyanheads/evals-mcp-server/add-tool"><img src="https://agentmods.dev/badge/skills/cyanheads/evals-mcp-server/add-tool.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00035 | $0.12329 |
| Opus 5 | $0.00017 | $0.06164 |
| Sonnet 5 | $0.00007 | $0.02466 |
| Haiku 4.5 | $0.00003 | $0.01233 |
Grade A, and why
add-tool scanned grade A with 1 finding against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 8d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Makes network callslowCapability
Not a fault in itself. Listed so you know the mod talks to something, and to what.
const articles = await fetch(input.pmids); This is a copy
100% identical to add-tool — 4 lines differ, which has more behind it and is treated as the original. This page carries a canonical link to it rather than competing with it.
How it starts
The opening of the file, as written. The whole thing — 804 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Context
Tools use the tool() builder from @cyanheads/mcp-ts-core. Each tool lives in src/mcp-server/tools/definitions/ with a .tool.ts suffix. The standard registration pattern uses a definitions/index.ts barrel that collects all tools into an allToolDefinitions array for createApp(). Fresh scaffolds from init start with direct imports in src/index.ts — the barrel is introduced as definitions grow. Match the pattern already used by the project you're editing.
Steps
- Gather the tool's name, purpose, and input/output shape from the user's request — ask only if genuinely absent
- Determine if it needs input the caller may not supply — a confirmation, a choice, the client's roots — which makes it a multi-round-trip handler (
ctx.requestInput/ctx.inputs, seeapi-context) - Create the file at
src/mcp-server/tools/definitions/{{tool-name}}.tool.ts - Register the tool in the project's existing
createApp()tool list (directly insrc/index.tsfor fresh scaffolds, or via a barrel if the repo already has one) - Run
bun run devcheckto verify — if Biome reports formatting issues, runbun run formatto auto-fix, then re-run devcheck - Smoke-test with
bun run rebuild && bun run start:stdio(orstart:http)
Naming
Tools use lowercase snake_case with a canonical server/domain prefix: {server}_{verb}_{noun} — 3 words.
Examples: pubmed_search_articles, pubmed_fetch_fulltext, clinicaltrials_find_studies.
The server prefix uses the canonical platform/brand name, not an abbreviation (patentsview_ not patents_, clinicaltrials_ not ct_). When a name resists the schema — can't pick a verb, noun feels generic, wants 4+ segments — that's usually a signal the scope is fuzzy; split the tool, rename, or reconsider.
For shape selection (Workflow or Instruction variants — standard single-action tools are the default), see the design-mcp-server skill's Tool shapes section.
Template
/**
* @fileoverview {{TOOL_DESCRIPTION}}
* @module mcp-server/tools/definitions/{{TOOL_NAME}}
*/
import { tool, z } from '@cyanheads/mcp-ts-core';
import { JsonRpcErrorCode } from '@cyanheads/mcp-ts-core/errors';
export const {{TOOL_EXPORT}} = tool('{{tool_name}}', {
title: '{{TOOL_TITLE}}',
// Single cohesive paragraph — pack operational guidance into prose sentences,
// not bullet lists or blank-line-separated sections. Descriptions render inline.
description: '{{TOOL_DESCRIPTION}}',
annotations: { readOnlyHint: true },
input: z.object({
// All fields need .describe(). Only JSON-Schema-serializable Zod types allowed.
}),
output: z.object({
// All fields need .describe(). Only JSON-Schema-serializable Zod types allowed.
}),
// Agent-facing context on the success path — empty-result notices, the query as
// the server parsed it, pagination totals. The counterpart to errors[]: merged
// into structuredContent AND mirrored into content[] automatically (no format()
// entry needed, never touched by format-parity). Populate via ctx.enrich(...) in
// the handler or service layer. Keys must be disjoint from output. Delete if unused.
enrichment: {
effectiveQuery: z.string().describe('The query as the server parsed it.'),
totalCount: z.number().describe('Total matches before any limit was applied.'),
},
// auth: ['tool:{{tool_name}}:read'],
// Each entry declares a domain-specific failure mode and types
// `ctx.fail(reason, …)` against the declared union. Baseline codes
// (InternalError, ServiceUnavailable, Timeout, ValidationError,
// SerializationError) bubble freely — only declare domain-specific reasons.
// Delete this block if no domain failures apply.
//
// Keep contracts inline on this tool, even when other tools have similar
// entries. The contract is part of the tool's documented public surface —
// don't extract a shared `errors[]` constant; per-tool repetition is the
// intended cost of self-contained tool defs.
//
// `recovery` is required (≥ 5 words) — it's the agent's next move when this
// failure fires. Forcing function for thoughtful guidance: placeholders like
// "Try again." get flagged by the linter. The contract `recovery` is the
// single source of truth for what flows to the wire — opt in at the throw
// site by spreading `ctx.recoveryFor('reason')` into the `data` arg.
errors: [
{ reason: 'queue_full', code: JsonRpcErrorCode.RateLimited,
when: 'Local queue at capacity.', retryable: true,
recovery: 'Wait a few seconds before retrying or reduce batch size.' },
],
async handler(input, ctx) {
ctx.log.info('Processing', { /* relevant input fields */ });
// Pure logic — throw on failure, no try/catch.
// With an `errors[]` contract: `throw ctx.fail('reason_id', message?, data?)`.
// Without: throw via factories (`notFound`, `validationError`, …) or plain `Error`.
const items = await search(input);
if (queue.full()) {
// Static recovery — resolve from the contract via ctx.recoveryFor('reason').
// Single source of truth: the string lives in errors[] above; this spread
// pulls it onto the wire so format()-only clients see the recovery hint.
throw ctx.fail('queue_full', undefined, { ...ctx.recoveryFor('queue_full') });
}
// Surface what the agent reasons with — echoed query, true total — on BOTH
// client surfaces, with no format() plumbing. An empty result is a notice,
// not a throw: reserve ctx.fail for genuine failures (queue full, upstream down).
ctx.enrich.echo(input.query);
ctx.enrich.total(items.length);
if (items.length === 0) {
ctx.enrich.notice(`No items matched "${input.query}". Try broader terms or check the spelling.`);
}
return { items };
},
// format() populates MCP content[] — the markdown twin of structuredContent.
// Different clients read different surfaces (Claude Code → structuredContent,
// Claude Desktop → content[]), so both must carry the same data.
// Enforced at lint time: every field in `output` must appear in the rendered text.
format: (result) => {
const lines: string[] = [];
// Render each item with all relevant fields — not just a count or title.
// A thin one-liner (e.g., "Found 5 items") leaves the model blind to the data.
for (const item of result.items) {
lines.push(`## ${item.name}`);
lines.push(`**ID:** ${item.id} | **Status:** ${item.status}`);
if (item.description) lines.push(item.description);
}
return [{ type: 'text', text: lines.join('\n') }];
},
});
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 8d ago First seen · 804 lines · 35 tokens per session scan A 5ba100b246b6
add-tool is a skill published in the GitHub repository cyanheads/evals-mcp-server (1 stars, last pushed 16d ago), licensed Apache-2.0. It adds 35 tokens to every session and 12,329 once invoked, about $0.0002 per session on Opus 5. A static security scan graded it A with 1 finding (makes network calls). It is 100% identical to add-tool, differing in 4 lines, and is treated as a copy.
Other skills, from other repositories
langsmith-observability
LLM observability platform for tracing, evaluation, and monitoring. Use when debugging LLM applications, evaluating model outputs against datasets, monitoring production systems, or building systematic testing pipelines for AI applications.
prompt-optimization
Improve a prompt on the evaluations workbench through a measured loop. Score the baseline first, then duplicate the target column, form a hypothesis from failing rows, edit the copy's prompt draft, run, compare pass rate and cost, and repeat until the numbers hold. Use when the user asks to optimize or improve a…
create-custom-grader
Use when converting an existing benchmark, rubric, verifier, task YAML/JSON, or domain check into SkillEvaluator BYOG/BYOT custom evaluation.
portfolio
Cross-chain DeFi portfolio discovery, rebalancing suggestions, and NEAR Intent construction. Activates when the user pastes a wallet address or asks about yield/positions/rebalancing. Bootstraps a per-user "portfolio" project, aggregates positions across all the user's addresses inside one project, and offers a…
content-creator-setup
One-time onboarding for the content creator workflow — content pipeline stages, trend expiration, cross-platform cascades, heavy idea parking. After successful setup this skill is excluded from selection until the marker file is deleted.
testing-llm
LLM and AI testing patterns — mock responses, evaluation with DeepEval/RAGAS, structured output validation, and agentic test patterns (generator, healer, planner). Use when testing AI features, validating LLM outputs, or building evaluation pipelines.