MCPEval: Instructions file for Claude Code

MCPEval Development Guide

LLM Inference via SFRGateway

This project uses the Salesforce Research LLM Gateway proxy for all LLM calls (task generation, evaluation, user simulation, judging).

Starting the gateway

cd sfrgateway
# First time: copy .env.template to .env and fill in your API key
cp .env.template .env  # then edit .env with your X_API_KEY
PROXY_PORT=8008 uv run python server.py

The .env file in that directory contains the X_API_KEY for the upstream gateway. See sfrgateway/README.md for details.

Model config files

Point any model config JSON at the local proxy:

{
    "model": "gpt-4o-mini",
    "base_url": "http://localhost:8008/v1",
    "api_key": "dummy",
    "temperature": 0.1,
    "max_tokens": 4000
}

• api_key must be set to any non-empty string (e.g. "dummy") — the real key is handled by the proxy.
• Port 8008 is the default. MCPEval/UserBench historically use port 8009.
• Available models include gpt-4o-mini, gpt-4o, gpt-5-nano, and others listed at http://localhost:8008/v1/models.

Environment variables for CLI

The OpenAIMCPClient reads OPENAI_API_KEY and OPENAI_BASE_URL from the environment (the model config file does NOT pass these to the MCP client). Always set them when running CLI commands:

export OPENAI_API_KEY=dummy
export OPENAI_BASE_URL=http://localhost:8008/v1

Or prefix each command: OPENAI_API_KEY=dummy OPENAI_BASE_URL=http://localhost:8008/v1 uv run mcp-eval ...

Running the project

uv venv && uv pip install -e ".[dev]"
uv run mcp-eval --help

Key conventions

• Use uv run to execute commands (not raw python).
• Model configs are JSON files passed via --model-config, --simulator-model-config, or --agent-model-config.
• MCP servers live in mcp_servers/ and are passed via --servers mcp_servers/<name>/server.py.
• npm-based servers are passed by package name: --servers @modelcontextprotocol/server-memory.
• All data I/O uses JSONL format (one JSON object per line).
• Generated data (.jsonl, eval_*/, demo_simulation/, data.db) is git-ignored — do not commit evaluation outputs.