create-mcp-evaluation

MCP Server Evaluation Creator

Create comprehensive evaluations to test whether LLMs can effectively use your MCP server to answer realistic, complex questions using only the tools provided.

Overview

The quality of an MCP server is measured by how well its implementations (input/output schemas, descriptions, functionality) enable LLMs with no other context to answer realistic and difficult questions.

Process

Step 1: Tool Inspection

1. List all tools available in the MCP server
2. Understand input/output schemas, descriptions, and annotations
3. Identify read-only vs. write operations
4. Note pagination capabilities and limits

Step 2: Content Exploration

1. Use READ-ONLY tools to explore available data
2. Identify stable data points that won't change over time
3. Map relationships between resources (projects → issues → comments → users)
4. Note interesting patterns, edge cases, and complex relationships

Step 3: Question Design

Create 10 evaluation questions following these requirements:

Core Rules

• Questions MUST be independent (no dependency on other answers)
• Questions MUST require ONLY read-only, non-destructive operations
• Questions MUST be realistic — tasks humans with LLM assistance would care about
• Each answer MUST be a single, verifiable value (string comparison)
• Answers MUST be stable (won't change over time)

Complexity Guidelines

• Require multiple tool calls (potentially dozens)
• Multi-hop: answer depends on chaining information from multiple queries
• Require deep exploration, not surface-level keyword search
• Use synonyms and paraphrases, not direct keywords from target content
• May require extensive pagination through results
• Should stress-test tool return values across data modalities

Answer Diversity Cover diverse answer types:

• Names (user, project, group)
• IDs (project ID, issue IID)
• URLs and paths
• Timestamps and dates (specify format in question)
• Counts and quantities
• Boolean (True/False)
• Status values