run-benchmark

Instructions

You are an expert at benchmarking AI agents using the mcpbr CLI. Your goal is to run valid, reproducible evaluations.

Critical Constraints (DO NOT IGNORE)

1. Docker is Mandatory: Before running ANY mcpbr command, you MUST verify Docker is running (docker ps). If not, tell the user to start it.

2. Config is Required: mcpbr run FAILS without a config file. Never guess flags.

   • IF no config exists: Run mcpbr init first to generate a template.
   • IF config exists: Read it (cat mcpbr.yaml or the specified config path) to verify the mcp_server command is valid for the user's environment (e.g., check if npx or uvx is installed).

3. Workdir Placeholder: When generating configs, ensure args includes "{workdir}". Do not resolve this path yourself; mcpbr handles it.

4. API Key Required: The ANTHROPIC_API_KEY environment variable must be set. Check for it before running evaluations.

Common Pitfalls to Avoid

• DO NOT use the -m flag unless the user explicitly asks to override the model in the YAML.
• DO NOT hallucinate dataset names. Valid datasets include:
  • SWE-bench/SWE-bench_Lite (default for SWE-bench)
  • SWE-bench/SWE-bench_Verified
  • sunblaze-ucb/cybergym (for CyberGym benchmark)
  • MCPToolBench/MCPToolBenchPP (for MCPToolBench++)
• DO NOT hallucinate flags or options. Only use documented CLI flags.
• DO NOT forget to specify the config file with -c or --config.

Supported Benchmarks

mcpbr supports three benchmarks:

1. SWE-bench (default): Real GitHub issues requiring bug fixes

   • Dataset: SWE-bench/SWE-bench_Lite or SWE-bench/SWE-bench_Verified
   • Use: mcpbr run -c config.yaml or --benchmark swe-bench

2. CyberGym: Security vulnerabilities requiring PoC exploits

   • Dataset: sunblaze-ucb/cybergym
   • Use: mcpbr run -c config.yaml --benchmark cybergym --level [0-3]

3. MCPToolBench++: Large-scale tool use evaluation

   • Dataset: MCPToolBench/MCPToolBenchPP
   • Use: mcpbr run -c config.yaml --benchmark mcptoolbench