mcp-eval

MCP Evaluation Harness

Runs an A/B comparison of the code-index MCP server against the default Read/Grep/Glob baseline on the same task, then reports tokens, tool calls, latency, and a judge-scored quality verdict.

Preconditions

Before doing anything else, verify all three:

1. The code-index MCP is connected — mcp__code-index__* tools must be visible in the current toolset. Check with /mcp if unsure.
2. .claude/index.db exists and was modified within the last 24 hours. Run ls -la .claude/index.db and check mtime.
3. The task is read-only. Phrasings like "find", "where", "explain", "trace", "what calls", "how does" are fine. Anything that says "edit", "fix", "add", "implement", "rename", "refactor" is NOT — running the same edit twice in parallel corrupts the working tree.

If any precondition fails, stop and tell the user which one. Do not proceed with workarounds.

Procedure

Step 1 — Lock the task prompt

Capture the user's task verbatim into $TASK. Do not rephrase, expand, or "improve" it. Both agents must receive byte-identical wording, or the comparison is invalid.

Step 2 — Spawn both research agents in parallel

Issue both Task tool calls in a single assistant message so they run concurrently. Sequential runs double wall time and skew the latency numbers.

• Agent A: subagent_type: "eval-with-mcp", prompt: $TASK
• Agent B: subagent_type: "eval-without-mcp", prompt: $TASK

These sub-agents are defined in .claude/agents/eval-with-mcp.md and .claude/agents/eval-without-mcp.md. Their tool restrictions are enforced at the framework level — do not pass extra tools.

Step 3 — Extract metrics from each result

Each sub-agent terminates its response with a fenced JSON block:

{
  "tool_calls": <int>,
  "files_inspected": ["path1", "path2"],
  "answer": "<final answer>",
  "confidence": 0.0
}

Parse this JSON. Then from the Task tool's metadata, capture:

• total_input_tokens (input + cache_read summed)
• total_output_tokens
• wall_time_seconds