Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add greynewell/mcpbr --skill benchmark-swe-litegit clone --depth 1 https://github.com/greynewell/mcpbrWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/greynewell/mcpbr/benchmark-swe-lite)<a href="https://agentmods.dev/skills/greynewell/mcpbr/benchmark-swe-lite"><img src="https://agentmods.dev/badge/skills/greynewell/mcpbr/benchmark-swe-lite/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/greynewell/mcpbr/benchmark-swe-lite"><img src="https://agentmods.dev/badge/skills/greynewell/mcpbr/benchmark-swe-lite.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00020 | $0.00771 |
| Opus 5 | $0.00010 | $0.00385 |
| Sonnet 5 | $0.00004 | $0.00154 |
| Haiku 4.5 | $0.00002 | $0.00077 |
Grade A, and why
swe-bench-lite scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 9d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 103 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Instructions
This skill provides a streamlined way to run the SWE-bench Lite benchmark with pre-configured defaults.
What This Skill Does
This skill runs a quick SWE-bench Lite evaluation with:
- 5 sample tasks (configurable)
- Verbose output for visibility
- Results saved to
results.json - Report saved to
report.md
Prerequisites Check
Before running, verify:
-
Docker is running:
docker ps -
API key is set:
echo $ANTHROPIC_API_KEY -
Config file exists:
- Check for
mcpbr.yamlin the current directory - If missing, run
mcpbr initto generate it
- Check for
Default Command
The default command for SWE-bench Lite:
mcpbr run -c mcpbr.yaml --dataset SWE-bench/SWE-bench_Lite -n 5 -v -o results.json -r report.md
Customization Options
Users can customize the run by modifying:
- Sample size: Change
-n 5to any number (or remove for full dataset) - Config file: Change
-c mcpbr.yamlto point to a different config - Verbosity: Use
-vvfor very verbose output - Output files: Change
results.jsonandreport.mdto different paths
Example Variations
Minimal quick test (1 task)
mcpbr run -c mcpbr.yaml -n 1 -v
Full evaluation (all ~300 tasks)
mcpbr run -c mcpbr.yaml --dataset SWE-bench/SWE-bench_Lite -v -o results.json
MCP-only (skip baseline)
mcpbr run -c mcpbr.yaml -n 5 -M -v -o results.json
Specific tasks
mcpbr run -c mcpbr.yaml -t astropy__astropy-12907 -t django__django-11099 -v
Expected Runtime & Cost
For 5 tasks with default settings:
- Runtime: 15-30 minutes (depends on task complexity)
- Cost: $2-5 (depends on task complexity and model used)
What to Do If It Fails
- Docker not running: Start Docker Desktop
- API key missing: Set with
export ANTHROPIC_API_KEY="sk-ant-..." - Config missing: Run
mcpbr initto generate default config - Config invalid: Check that
{workdir}placeholder is in theargsarray - MCP server fails: Test the server command independently
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 9d ago First seen · 103 lines · 20 tokens per session scan A 7045de1ff87c
swe-bench-lite is a skill published in the GitHub repository greynewell/mcpbr (10 stars, last pushed 4mo ago), licensed MIT. It adds 20 tokens to every session and 771 once invoked, about $0.0001 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
api-testing
Testing patterns for MCP tool/resource handlers using createMockContext and Vitest. Covers mock context options, handler testing, McpError assertions, format testing, Vitest config setup, and test isolation conventions.
add-test
Scaffold a test file for an existing tool, resource, or service. Use when the user asks to add tests, improve coverage, or when a definition exists without a matching test file.
Home Security AI Benchmark
LLM & VLM evaluation suite for home security AI applications.
SmartHome Video Anomaly Benchmark
VLM evaluation suite for video anomaly detection in smart home camera footage.
email-formatting
Markdown formatting conventions for email summary documents — heading depth, list style, line length, emoji policy, and a mandatory provenance footer. Read this when producing a markdown report that summarizes one or more email messages so the output matches the project's house style.
json-schema-emails
Canonical shape of the .workflow/emails/emails.json file passed between the fetch and summarize states — required fields (sender, recipient, subject, date, body), types, and field semantics. Read this whenever you write or read emails.json so producer and consumer agree on the shape.