mcp-eval

MCP Server Evaluation Guide (Phoenix)

Overview

The measure of an MCP server's quality is NOT how comprehensively it implements tools, but how well those tools (schemas, docstrings, return shapes) let an LLM with NO other context answer realistic, difficult questions. This skill describes a reusable Arize Phoenix evaluation harness that:

1. Spins up a LangChain agent connected to the MCP server (over stdio).
2. Feeds it a dataset of natural-language questions with known answers.
3. Scores each answer with LLM-as-judge evaluators and logs traces to Phoenix.

Use this after the server is built (it complements the mcp-builder skill, which produces the evaluation.xml question set this harness consumes).

---

Module Layout

The harness lives in eval/phoenix/ at the repo root:

eval/phoenix/
├── agent.py                   # The <Server>Agent class (launches the MCP server over stdio)
├── create_dataset.py          # CLI: upload a CSV dataset to Phoenix
├── run_experiment.py          # CLI: run an experiment (agent + judges) against a dataset
├── datasets.yaml              # Dataset registry (name -> csv_path, input/output keys, description)
├── README.md                  # Usage docs for this specific server
├── judges/                    # LLM-as-judge evaluators
│   ├── __init__.py            # Re-exports each judge for `from judges import ...`
│   ├── correctness_judge.py   # match_expected_response (compares to ground truth)
│   └── relevance_judge.py     # check_answer_scope (in-scope vs. out-of-scope)
├── prompts/                   # Agent system prompts, one per version
│   ├── system_prompt_v1.txt   # Basic domain analyst
│   └── system_prompt_v2.txt   # + scope boundaries
└── datasets/                  # Test datasets (CSV files)
    └── <name>/examples/<name>.csv

Naming: Name the agent class after the server (e.g. HydroAgent, NIHReporterAgent) and name datasets <domain>-eval-<n> (e.g. hydro-eval-0).