Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/sjarmak/agent-workflows/mcp-evalnpx skills add sjarmak/agent-workflows --skill mcp-evalgit clone --depth 1 https://github.com/sjarmak/agent-workflowsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/sjarmak/agent-workflows/mcp-eval)<a href="https://agentmods.dev/skills/sjarmak/agent-workflows/mcp-eval"><img src="https://agentmods.dev/badge/skills/sjarmak/agent-workflows/mcp-eval.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00029 | $0.02904 |
| Opus 5 | $0.00015 | $0.01452 |
| Sonnet 5 | $0.00006 | $0.00581 |
| Haiku 4.5 | $0.00003 | $0.00290 |
Grade A, and why
mcp-eval scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 4d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 324 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Evaluate an MCP tool's agentic usability through structured testing and meta-reflection. Spawns parallel agents to run test scenarios against a tool, then runs a separate reflection pass to capture the agent's subjective experience. Produces a scored report with actionable improvement recommendations.
Arguments
$ARGUMENTS — format: [tool_name] [repo_context] where tool_name is required (e.g., "deepsearch", "keyword_search") and repo_context is an optional repository to scope tests against (e.g., "github.com/sourcegraph/sourcegraph")
Parse Arguments
Extract:
- tool_name: the MCP tool to evaluate (required). Match against available Sourcegraph MCP tools: deepsearch, keyword_search, nls_search, commit_search, diff_search, find_references, go_to_definition, compare_revisions, list_files, list_repos, read_file, deepsearch_read, get_contributor_repos
- repo_context: optional repository to use for test queries (default: "github.com/sourcegraph/sourcegraph")
If tool_name is missing or doesn't match a known tool, list available tools and ask the user to pick one.
Phase 1: Tool Profile
Before testing, build a profile of the tool under evaluation:
- Read the tool's description and parameter schema (use ToolSearch if needed)
- Document:
- Purpose (as stated in description)
- Parameters (required vs optional, types, constraints)
- Expected output format
- Stated use cases (from description/examples)
- Stated anti-patterns (when NOT to use)
- Overlap with other tools (which tools could serve similar purposes)
Present the tool profile to the user and confirm before proceeding. Adjust if the user gives feedback.
Phase 2: Generate Test Scenarios
Based on the tool profile, generate 8 test scenarios covering these dimensions. Tailor the specific queries to the tool being tested — these are templates, not literal tests:
Scenario 1: Broad Conceptual Query
A vague, exploratory question that tests whether the tool handles ambiguity well. Example for deepsearch: "How does authentication work in this codebase?" Example for keyword_search: "authentication"
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 4d ago First seen · 324 lines · 29 tokens per session scan A 8515ec3e93b2
mcp-eval is a skill published in the GitHub repository sjarmak/agent-workflows (9 stars, last pushed 1mo ago), licensed MIT. It adds 29 tokens to every session and 2,904 once invoked, about $0.0001 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
speckit.clarify
Skill "speckit.clarify" from caipe-io/ai-platform-engineering, covering user input and outline.
speckit.specify
Skill "speckit.specify" from caipe-io/ai-platform-engineering, covering user input, outline, quick guidelines, section requirements and for ai generation.
release-docs
Generate a combined release blog post for ai-platform-engineering. Produces a single docs/releases/YYYY-MM-DD-release-X-Y-Z.md file containing release notes and the upgrade guide (migration guide) inline. Use when cutting a release, when a user asks "what changed in 0.4.x", or when upgrading their values.yaml to a new…
update-docs
Audit and update all documentation moving parts for ai-platform-engineering. Checks release blog posts, features page, agent docs, homepage version strings, Docusaurus version config, and sidebar completeness. Fixes what is stale and reports what needs manual attention. Use after cutting a release, adding a new agent…
local-integration-testing
Run end-to-end integration tests with all 15 agents and supervisor in local Docker Compose dev environment. Validates agent discovery, multi-agent routing, checkpoint persistence, and cross-agent follow-up conversations.
docker-compose-first-install
Validate and repair the OSS first-install Docker Compose path. Use when editing docker-compose.yaml, docker-compose.dev.yaml, .env.example, release image tags, Compose profiles, Keycloak/OpenFGA/RAG defaults, or first-launch UI behavior for local all-in-one installs.