mcp-tester-guide

MCP Server Tester — Reference Guide

What This Framework Does

@gleanwork/mcp-server-tester is a Playwright-based testing and evaluation framework for Model Context Protocol (MCP) servers. It provides:

• Playwright fixtures for automated MCP tool testing
• Data-driven eval datasets with optional LLM-as-a-judge scoring
• MCP host simulation via real LLM providers (Anthropic, OpenAI, Google, etc.)
• A custom Playwright reporter with an interactive UI

Requires Node.js >= 22.0.0 and Playwright.

Import Paths

The package exposes four entry points:

// Main library — types, validators, eval runner, judge, config
import { ... } from '@gleanwork/mcp-server-tester';

// Playwright test fixtures — test() and expect() with MCP matchers
import { test, expect } from '@gleanwork/mcp-server-tester/fixtures/mcp';

// Auth-specific fixtures — for OAuth and token auth testing
import { test } from '@gleanwork/mcp-server-tester/fixtures/mcpAuth';

// Custom Playwright reporter
import mcpReporter from '@gleanwork/mcp-server-tester/reporters/mcpReporter';

Always import test and expect from @gleanwork/mcp-server-tester/fixtures/mcp in test files. The expect from the main entry does NOT include MCP matchers.

Transport Configuration

Configure mcpConfig in playwright.config.ts under project.use:

stdio transport

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  projects: [
    {
      name: 'my-mcp-server',
      use: {
        mcpConfig: {
          transport: 'stdio',
          command: 'node',
          args: ['./dist/server.js'],
          env: { DEBUG: 'true' },
        },
      },
    },
  ],
});

HTTP (Streamable HTTP / SSE) transport

{
  mcpConfig: {
    transport: 'http',
    url: 'http://localhost:3000/mcp',
    headers: { 'Authorization': 'Bearer token' },
  },
}

All 11 Matchers

Use these with expect(result) after calling mcp.callTool():