write-mcp-eval

Write MCP Eval Datasets

Generate data-driven eval datasets for MCP server tools using @gleanwork/mcp-server-tester. Eval datasets are JSON files containing test cases that are loaded and executed by the eval runner.

When to Use Evals vs Inline Tests

Use eval datasets when:

• You have many similar test cases for the same tool (10+ cases)
• Test data changes more frequently than test logic
• Non-engineers need to add or review test cases
• You want to track accuracy over time with baselines

Use inline Playwright tests (see write-mcp-test skill) when:

• You need complex setup or teardown logic
• Tests have unique validation requirements
• You're testing tool interactions or multi-step flows

Step 1 — Create the Eval Dataset JSON

Minimal dataset

{
  "name": "search-tool-evals",
  "cases": [
    {
      "id": "basic-query",
      "toolName": "search",
      "args": { "query": "quarterly planning" },
      "expect": {
        "containsText": "quarterly"
      }
    }
  ]
}

Full dataset structure

{
  "name": "search-tool-evals",
  "description": "Regression tests for the search tool",
  "metadata": {
    "version": "1.0",
    "author": "team-name"
  },
  "cases": [
    {
      "id": "basic-query",
      "description": "Simple keyword search returns relevant results",
      "toolName": "search",
      "args": { "query": "quarterly planning" },
      "tags": ["regression", "search"],
      "expect": {
        "containsText": ["quarterly", "planning"],
        "isError": false,
        "responseSize": { "maxBytes": 50000 }
      }
    }
  ]
}

Step 2 — Write Expectations

Each case's expect block maps directly to a Playwright matcher. Combine multiple expectations — all must pass.

Text containment (containsText)

{
  "id": "text-check",
  "toolName": "search",
  "args": { "query": "onboarding" },
  "expect": {
    "containsText": "onboarding"
  }
}

Multiple strings — all must be present:

"expect": {
  "containsText": ["onboarding", "guide", "new hire"]
}