field-test

Context

Unit tests (add-test skill) verify handler logic with mocked context. Field testing exercises the real HTTP transport with real JSON-RPC: starts the server, calls initialize, surfaces the catalog, runs inputs, and checks what a client actually sees. It catches what unit tests miss — awkward input shapes, unhelpful errors, missing format output, drift between structuredContent and content[], edge-case surprises.

Actively call the tools. Don't read code and guess.

Transport coverage

This skill drives an HTTP server because curl + JSON-RPC is the most reliable harness for shell-based agents. The same handlers run on both transports — only the framing differs — so HTTP exercises the full functional surface.

Stdio coverage is a boot check only — run this before Step 1. Run bun run rebuild && bun run start:stdio, confirm the startup logs look clean (banner, expected tool/resource counts, no errors/warnings, no missing-config gripes), then kill it. Pino logs go to stderr in stdio mode (stdout is reserved for JSON-RPC), so they print straight to the terminal when you run interactively. No need to call tools over stdio — the HTTP pass already covered handler behavior.

---

Steps

1. Start the server

Generate a 10-character alphanumeric ID (e.g. 9DJ73-K103L) and write the helper to /tmp/<project-name>-field-test-<ID>.sh. Use that exact path in every subsequent Bash call. Two agents in the same project tree must pick different IDs — that's what keeps their helper files, server logs, and call scratch from colliding.

The helper itself is stateless — every function takes the IDs it needs (server pid, url, port, MCP sid, server log path) as positional args. mcp_start prints them; the agent threads them through every later call. No env vars, no shared state files.

# Pick your ID — example below uses 9DJ73-K103L. Substitute your own.
# (Helper path also encodes the project name so /tmp/ stays grep-friendly.)
cat > /tmp/<project-name>-field-test-9DJ73-K103L.sh <<'HELPER_EOF'
#!/bin/bash
# Field-test helper: stateless wrappers around an MCP HTTP server + JSON-RPC
# session. Every function takes the IDs it needs as positional args — the agent
# threads pid/url/port/sid/log through each call rather than relying on a state
# file or env vars (the Bash tool wipes shell state between calls, and a
# pointer file would race the same way two agents race on shared state).
# See https://github.com/cyanheads/mcp-ts-core/issues/90, #144.
#
# Surfaces failures aggressively — field test is for finding things that fail,
# so the helper auto-tails logs and prints HTTP status/body on errors instead
# of swallowing them.

# Usage: mcp_start /path/to/server [startup-timeout-seconds]   (default: 30)
# Builds, starts the HTTP server in the background, waits for the listen line,
# and prints: ready pid=<n> url=<u> port=<n> log=<path>
# Capture these — every later helper takes them as args. Raise the timeout for
# servers that build a local index at boot.
mcp_start() {
  local dir="${1:-$PWD}"
  local timeout="${2:-30}"
  local build_log; build_log=$(mktemp /tmp/mcp-field-test-build.XXXXXX)
  echo "building $dir ..." >&2
  if ! (cd "$dir" && bun run rebuild) >"$build_log" 2>&1; then
    echo "BUILD FAILED — last 30 lines of $build_log:" >&2
    tail -30 "$build_log" >&2
    return 1
  fi
  rm -f "$build_log"
  local server_log; server_log=$(mktemp /tmp/mcp-field-test-server.XXXXXX)
  echo "starting server ..." >&2
  (cd "$dir" && bun run start:http) >"$server_log" 2>&1 &
  local pid=$!
  local line=""
  local waited=0
  while [ "$waited" -lt "$((timeout * 4))" ]; do
    line=$(grep -Eo 'listening at http://[^" ]+/mcp' "$server_log" | head -1)
    [ -n "$line" ] && break
    if ! kill -0 "$pid" 2>/dev/null; then
      echo "server exited during startup — last 30 lines of $server_log:" >&2
      tail -30 "$server_log" >&2
      rm -f "$server_log"
      return 1
    fi
    sleep 0.25
    waited=$((waited + 1))
  done
  if [ -z "$line" ]; then
    echo "server failed to start within ${timeout}s — last 30 lines of $server_log:" >&2
    tail -30 "$server_log" >&2
    kill "$pid" 2>/dev/null
    rm -f "$server_log"
    return 1
  fi
  local url="${line#listening at }"
  local port; port=$(echo "$url" | sed -E 's|.*:([0-9]+)/.*|\1|')
  echo "ready pid=$pid url=$url port=$port log=$server_log"
}

# Usage: mcp_init <url>
# Runs `initialize`, sends `notifications/initialized`, prints:
#   ready sid=<id> protocol=<negotiated-version>
# A negotiated version older than the requested one means the server capped it
# — note that in the report; you are then testing an older protocol than a
# current client would use.
mcp_init() {
  local url="$1"
  [ -z "$url" ] && { echo "usage: mcp_init <url>" >&2; return 1; }
  local want="${MCP_FIELD_TEST_PROTOCOL:-2025-11-25}"
  local hdr; hdr=$(mktemp)
  local body_file; body_file=$(mktemp)
  local code
  code=$(curl -sS -D "$hdr" -o "$body_file" -w '%{http_code}' -X POST "$url" \
    -H "Content-Type: application/json" \
    -H "Accept: application/json, text/event-stream" \
    -d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"initialize\",\"params\":{\"protocolVersion\":\"$want\",\"capabilities\":{},\"clientInfo\":{\"name\":\"field-test\",\"version\":\"1.0.0\"}}}")
  local sid; sid=$(grep -i '^mcp-session-id:' "$hdr" | awk '{print $2}' | tr -d '\r\n')
  if [ -z "$sid" ]; then
    echo "init failed — HTTP $code, no Mcp-Session-Id header returned" >&2
    echo "--- response body ---" >&2
    cat "$body_file" >&2
    echo "--- response headers ---" >&2
    cat "$hdr" >&2
    rm -f "$hdr" "$body_file"
    return 1
  fi
  local got; got=$(sed -n 's/^data: //p' "$body_file" | grep -o '"protocolVersion":"[^"]*"' | head -1 | cut -d'"' -f4)
  [ -z "$got" ] && got=$(grep -o '"protocolVersion":"[^"]*"' "$body_file" | head -1 | cut -d'"' -f4)
  curl -sS -X POST "$url" \
    -H "Content-Type: application/json" \
    -H "Accept: application/json, text/event-stream" \
    -H "Mcp-Session-Id: $sid" \
    -d '{"jsonrpc":"2.0","method":"notifications/initialized"}' >/dev/null
  rm -f "$hdr" "$body_file"
  echo "ready sid=$sid protocol=${got:-unknown} requested=$want (HTTP $code)"
}

# Usage: mcp_call <url> <sid> <method> [JSON_PARAMS]
# Prints the JSON-RPC response. SSE framing is stripped when present, and only
# the reply is emitted (a single POST can also carry progress notifications, so
# emitting every event would break `| jq .result`). A transport failure or an
# HTTP >= 400 prints the details and returns non-zero — it never returns 0 with
# empty output. Pipe to `jq`.
mcp_call() {
  local url="$1"; local sid="$2"; local method="$3"; local params="${4:-}"
  [ -z "$url" ] || [ -z "$sid" ] || [ -z "$method" ] && { echo "usage: mcp_call <url> <sid> <method> [params]" >&2; return 1; }
  local body
  if [ -z "$params" ]; then
    body=$(printf '{"jsonrpc":"2.0","id":%d,"method":"%s"}' "$RANDOM" "$method")
  else
    body=$(printf '{"jsonrpc":"2.0","id":%d,"method":"%s","params":%s}' "$RANDOM" "$method" "$params")
  fi
  local resp_file; resp_file=$(mktemp)
  local code curl_rc
  code=$(curl -sS -o "$resp_file" -w '%{http_code}' -X POST "$url" \
    -H "Content-Type: application/json" \
    -H "Accept: application/json, text/event-stream" \
    -H "Mcp-Session-Id: $sid" \
    -d "$body")
  curl_rc=$?
  if [ "$curl_rc" -ne 0 ] || [ -z "$code" ] || [ "$code" = "000" ]; then
    echo "TRANSPORT FAILURE calling $method — curl exit $curl_rc, http_code '${code:-none}'." >&2
    echo "Server not reachable at $url (check it is still running: mcp_log <log>)." >&2
    rm -f "$resp_file"
    return 1
  fi
  if [ "$code" -ge 400 ]; then
    echo "HTTP $code from $method — response:" >&2
    cat "$resp_file" >&2
    rm -f "$resp_file"
    return 1
  fi
  local sse; sse=$(sed -n 's/^data: //p' "$resp_file")
  if [ -n "$sse" ]; then
    local reply; reply=$(printf '%s\n' "$sse" | grep -E '"(result|error)"')
    printf '%s\n' "${reply:-$sse}"
  else
    cat "$resp_file"
  fi
  rm -f "$resp_file"
}

# Usage: mcp_log <server-log-path> [N]   (default: 50 lines)
# Tail the per-server log printed by mcp_start. Useful when a call surprises
# you — pino startup banner, definition lint diagnostics, request handler
# errors, upstream calls, and rate-limit warnings all land here.
mcp_log() {
  local log="$1"; local n="${2:-50}"
  [ -z "$log" ] && { echo "usage: mcp_log <log-path> [n]" >&2; return 1; }
  tail -n "$n" "$log"
}

# Usage: mcp_stop <pid> [server-log-path] [port]
# Kills the background server and the `bun run` child that actually holds the
# port (SIGKILL is not forwarded, so the child must be signalled directly or it
# survives as an orphaned listener). Pass the port from mcp_start to have the
# stop confirmed against the socket rather than against the wrapper PID.
# Removes the server log if a path is given.
mcp_stop() {
  local pid="$1"; local log="${2:-}"; local port="${3:-}"
  [ -z "$pid" ] && { echo "usage: mcp_stop <pid> [log-path] [port]" >&2; return 1; }
  local kids; kids=$(pgrep -P "$pid" 2>/dev/null)
  kill "$pid" $kids 2>/dev/null
  for _ in $(seq 1 12); do
    kill -0 "$pid" 2>/dev/null || break
    sleep 0.25
  done
  if kill -0 "$pid" 2>/dev/null; then
    echo "PID $pid didn't exit on SIGTERM — sending SIGKILL"
    kill -9 "$pid" $kids 2>/dev/null
    sleep 0.5
  fi
  local held=""
  [ -n "$port" ] && held=$(lsof -ti tcp:"$port" 2>/dev/null | tr '\n' ' ')
  if [ -n "$held" ]; then
    echo "WARNING: port $port still held by PID(s) $held after stopping $pid — kill those before re-running"
  elif kill -0 "$pid" 2>/dev/null; then
    echo "WARNING: PID $pid still alive after SIGKILL"
  else
    echo "stopped pid=$pid${port:+ (port $port free)}"
  fi
  [ -n "$log" ] && rm -f "$log"
  return 0
}
HELPER_EOF

. /tmp/<project-name>-field-test-9DJ73-K103L.sh
mcp_start /absolute/path/to/server   # replace with the target server