mcp-server-tester: Instructions file for Codex

AGENTS.md

This file provides guidance to coding agents when working with code in this repository.

Project Overview

@gleanwork/mcp-server-tester is a Playwright-based testing and evaluation framework for Model Context Protocol (MCP) servers. It provides Playwright fixtures for automated testing and data-driven eval datasets with optional LLM-as-a-judge scoring.

Common Commands

# Build (includes UI reporter build)
npm run build

# Unit tests (Vitest)
npm test                    # Run all unit tests
npm run test:watch          # Watch mode
npm test -- src/mcp/clientFactory.test.ts  # Run single test file
npm test -- -t "creates client"            # Run tests matching pattern

# Integration tests (Playwright)
npm run test:playwright

# Code quality
npm run typecheck           # TypeScript validation
npm run lint                # ESLint
npm run lint:fix            # Auto-fix lint issues
npm run format              # Prettier formatting
npm run format:check        # Check formatting

Architecture

Core Modules (src/)

• config/ - MCPConfig types and Zod validation for stdio/HTTP transports, protocolMatrix()
• mcp/ - Client factory (createMCPClientForConfig), fixtures (MCPFixtureApi), protocol selection (protocol.ts), wire tap, and response normalization. Built on the MCP TypeScript SDK v2 (@modelcontextprotocol/client). mcp/connection.ts is the one record of what MST knows about a client it created (requested protocol, connection target for raw probes, wire tap, owned undici agent); read it with connectionOf(client), and add new per-connection facts there rather than in a new side table
• skills/ - Agent Skills over MCP (SEP-2640): wire schemas, entry validation, and a skills client (the SDK has no skills API yet)
• auth/ - OAuth 2.1 with PKCE (PlaywrightOAuthClientProvider) and static token utilities. auth/grants/ holds connectors and refresh grants: connector lookup and grant grouping (connectors.ts), sign-in, refresh and revocation (grants.ts), and the built-in credential store mst/credential-store/local (localStore.ts). A run stages and renews connector tokens in evals/connectorServers.ts
• assertions/ - Unified assertion architecture (see below)
• evals/ - Dataset types, loader, and runner (uses validators internally). evals/caseExecution.ts is the only place a case runs: every path (the mst client on a Playwright test's connection, an eval's client, a custom executeCase) returns a typed CaseExecution (completed or failed), and the runner reads its fields instead of inspecting response. caseExecution.golden.test.ts pins the resulting EvalCaseResult for every path. evals/grading.ts grades a case's assertions block: it decides once whether the evidence can support tool-call assertions (toolEvidenceGap), builds the toolCallTrace view, resolves judge settings (including the eval config's judges), then calls the validators. Desktop clients (Cowork, ChatGPT) run batches through evals/desktopBatch.ts (lease, per-case reset policy, failure diagnosis before a reset, client-native session ledger, redaction, cleanup) and share one MCP readiness rule (isMcpServerReady in evals/mcpReadiness.ts); a new desktop client supplies only prepare, one case, reset, an optional diagnose and dispose. macOS Swift controllers build and run through evals/nativeHelper.ts, which owns their environment policy
• llm/ - resolveLLMEndpoint(): the one place MST's own LLM calls (SDK client, judges) get their base URL and credential, including gateway bearer tokens and MST_LLM_AUTH_COMMAND. New LLM consumers resolve through it rather than reading *_API_KEY themselves. See docs/llm-gateways.md
• evals/runFormat.ts - The mst.run/v1 run directory: run IDs, paths, the writer and the Zod schemas schema/run/v1/ is generated from (npm run schema:generate)
• judge/ - LLM-as-a-judge. llmJudge.ts is the one core (prompt, verdict parser, output guard); each provider in JUDGE_PROVIDER_KINDS (anthropic, vertex-anthropic, anthropic-agent-sdk, openai, google) is a completion adapter mapped in judgeClient.ts. judge/agentic/ holds agentic judges (agenticJudge, agenticPairwiseJudge), which run an agent over a workspace of the trial's evidence on the claude-agent or codex runtime, and stagedPairwiseJudge (stagedJudge.ts), which runs a staged judge: a serialized bundle of files, prepare programs and agent stages. builtinJudges.ts has the built-in rubric judge
• proxy/ - The dry-run proxy (dryRunProxyServer()), which a connector's launch puts in front of a vendor server to let reads through and block writes, and simulated writes (simulatedWrites.ts), which answer blocked writes with a success reply when an eval config sets simulateWrites
• evals/environments/ - Where trials are collected (ADR 0004): the env kind's built-ins local (builtinEnvironments.ts) and docker (docker.ts: containers from the Linux Cowork image in docker/cowork-linux/, one per Docker context at a time, over the Docker CLI; dockerContainers.ts removes them when MST exits or its process is gone; mst env setup docker builds the image), the channel helper machineEnvironment (channel.ts), the mst.shard/v1 messages (protocol.ts), the coordinator's sharded collect (shardedCollect.ts), the worker behind the hidden mst collect (collect.ts), and the run-wide limits (limiter.ts)
• plugins/ - ESLint-style plugins: the Plugin shape and validation (plugin.ts), the process-wide extension table keyed by <namespace>/<kind>/<name> (names parsed and kind-checked in plugin.ts and extensions.ts), where each kind's lookup (next to its built-ins) adds them under bare names (extensions.ts), and the loader (loadPlugins.ts). evals/evalPlugins.ts loads an eval's plugins and checks it only references namespaces it loads. Domain terms are in CONTEXT.md; decisions in docs/adr/
• spec/ - Conformance check registry (checks/core.ts, checks/modern.ts, checks/skills.ts), raw probe channel, and cross-era checks
• reporters/ - The run report UI (React) and the MCP Playwright reporter, which reports evals only: each Playwright run's eval results are a mst.run/v1 run (a variant per project) with the same report mst run writes (evals/runReport.ts). Tests and conformance checks are in Playwright's own reporter. reporters/channel.ts owns every attachment the reporter reads (names, payload types, read-side Zod schemas). Write with attachReporterData(testInfo, { kind, data }), never testInfo.attach('mcp-...') directly
• cli/ - The CLI, shipped as mst and mcp-server-tester (the same binary): init, generate, login, token, auth, datasets, judges, run, grade, batch, cowork, open, and the hidden collect that a worker runs