mcp-server-testing

Test an MCP Server

Two different questions. Does it respond correctly? is testing. Can an agent accomplish a task with it? is evaluation. A server can pass the first completely and fail the second, and the second is the one users feel.

The Inspector

An interactive client that lists your tools, calls them with arguments you supply, and shows the raw protocol traffic.

# TypeScript
npm run build && npx @modelcontextprotocol/inspector ./dist/server.js

# Python
python -m py_compile your_server.py
npx @modelcontextprotocol/inspector -- python your_server.py

# Any executable
npx @modelcontextprotocol/inspector /path/to/server

If a tool does not appear in the Inspector, no agent will see it either. Check this before suspecting anything subtler.

Pre-Ship Checks

• Every tool has a description saying when to use it, not only what it does
• Errors return actionable text with isError: true
• I/O is async, and paginated wherever results can grow
• Nothing writes to stdout on a stdio server
• Annotations are declared, especially destructiveHint
• Input schemas carry constraints, so bad input fails at the boundary

Evaluation

Testing proves the server responds. Evaluation proves an agent can use it — and it is the only way to discover that a tool description is technically accurate and practically useless.

Write ten questions requiring real tool use, then answer each yourself so you hold ground truth. Each question should be:

• Independent — not reliant on another question's answer
• Read-only — no destructive operations
• Multi-step — several tool calls and some exploration
• Realistic — something a person would actually ask
• Verifiable — one clear answer, checkable by string comparison
• Stable — the answer will not drift next week

<evaluation>
  <qa_pair>
    <question>Which repository had the most merged pull requests last quarter, and how many?</question>
    <answer>servers, 47</answer>
  </qa_pair>
</evaluation>