mcp eval.instructions.md

---
applyTo: 'eval/**'
---

# Copilot Instructions for Storybook MCP Eval Harness

## Project Overview

This is an eval harness for testing AI coding agents' ability to build UI components using Storybook and MCP tools. The harness automates the process of running trials, executing agents with prompts, and grading the results through multiple quality metrics.

**Core Purpose**: Measure how effectively AI agents can use Storybook's MCP tools to build production-quality UI components.

## Architecture

### Harness Flow

1. **Prepare**: Create fresh Vite + React + Storybook project from template
2. **Execute**: Run AI agent (Claude Code CLI) with prompt and optional context
3. **Grade**: Run automated checks (build, typecheck, lint, tests, a11y)
4. **Report**: Generate metrics, save results, optionally upload to Google Sheets

### Key Components

- **CLI Interface**: Two modes:
  - `eval.ts` - Batch eval runner for running multiple trials across variants
  - `advanced-eval.ts` - Single trial runner with interactive CLI
- **Context System**: Five context modes (none, Storybook MCP - Dev, Storybook MCP - Docs, custom MCP server, extra prompts)
- **Agent Abstraction**: Pluggable agent implementations (currently Claude Code CLI)
- **Grading Pipeline**: Parallel execution of multiple quality checks
- **Hooks System**: Lifecycle hooks for custom trial logic
- **Telemetry**: Optional results upload to Google Sheets for tracking

### File Structure

```
eval/
├── eval.ts                          # Batch eval runner (multiple trials)
├── advanced-eval.ts                 # Single trial runner (interactive)
├── types.ts                         # Core types and schemas
├── lib/
│   ├── collect-args.ts              # Interactive CLI argument collection (for advanced-eval)
│   ├── show-help.ts                 # Help text formatting
│   ├── generate-prompt.ts           # Combines prompt parts
│   ├── prepare-trial.ts             # Project template setup
│   ├── teardown-trial.ts            # Trial cleanup (stop dev servers, etc.)
│   ├── storybook-dev-server.ts      # Storybook dev server management
│   ├── agents/
│   │   └── claude-code-cli.ts       # Claude Code CLI agent implementation
│   ├── eval/                        # Batch eval runner logic
│   │   ├── collect-eval-args.ts     # Collect args for batch eval
│   │   ├── run-eval.ts              # Main eval orchestration
│   │   ├── types.ts                 # Eval-specific types
│   │   └── progress-ui.ts           # Progress display
│   └── graders/
│       ├── grade.ts                 # Main grading orchestrator
│       ├── prepare-grading.ts       # Install test dependencies
│       ├── build.ts                 # Vite build check
│       ├── typecheck.ts             # TypeScript checking
│       ├── lint.ts                  # ESLint execution
│       ├── test-stories.ts          # Vitest + a11y testing
│       ├── environment.ts           # Git branch/commit tracking
│       └── save-to-sheets.ts        # Google Sheets upload
├── variant-configs/                 # Variant configs (variants, context combinations)
├── tasks/                           # Task definitions
│   └── {number}-{name}/
│       ├── prompt.md                # Main prompt
│       ├── hooks.ts                 # Optional lifecycle hooks
│       ├── manifests/               # Optional manifest files directory
│       │   ├── components.json      # Component manifest for @storybook/mcp
│       │   └── docs.json            # Optional docs manifest for @storybook/mcp
│       ├── mcp.config.json          # Optional MCP server config
│       ├── *.md                     # Optional additional context
│       ├── pre-grade/               # Optional files to copy before grading
│       ├── {hook-name}/             # Optional hook directories (see Lifecycle Hooks)
│       └── trials/                  # Generated trial runs
└── templates/
    ├── project/                     # Base Vite + React template
    └── grading/                     # Test/lint configs
```

### Context Modes

The harness supports five distinct context types:

1. **No Context** (`--no-context`):
   - Agent uses only built-in tools
   - Tests baseline agent capabilities

2. **Storybook MCP - Dev** (`--context storybook-dev`):
   - Sets up Storybook in the trial project with `@storybook/addon-mcp`
   - Starts a local Storybook dev server
   - Configures the MCP server endpoint (`/mcp`) for the agent
   - Best for testing agents in a live development environment
   - Includes automatic setup and teardown of the dev server

3. **Storybook MCP - Docs** (`--context components.json`):
   - Provides component documentation via `@storybook/mcp` package
   - Uses stdio transport with `packages/mcp/bin.ts`
   - Best for testing agents with library/component documentation
   - This uses the Storybook MCP server, not a custom MCP server

4. **MCP Server** (`--context mcp.config.json` or inline JSON):
   - Custom MCP server configuration (HTTP or stdio)
   - Supports multiple named servers
   - Flexible for testing different MCP tool combinations
   - Use this for fully custom MCP servers; use components.json for Storybook MCP - Docs

5. **Extra Prompts** (`--context extra-prompt-01.md,extra-prompt-02.md`):
   - Appends additional markdown files to main prompt
   - Useful for providing supplementary instructions
   - Keeps main prompt clean while testing variations

## Development Workflow

### Prerequisites

- Node.js 24+ (see root `.nvmrc`)
- pnpm 10.19.0+ (monorepo root enforces this)
- Claude Code CLI: `npm install -g claude-code`

### Running Evaluations

**Batch eval mode (recommended for comparison testing):**
```bash
cd eval
node eval.ts
```

**Single trial mode (interactive):**
```bash
cd eval
node advanced-eval.ts
```

**Non-interactive single trial:**
```bash
node advanced-eval.ts --agent claude-code --context components.json --upload-id batch-1 --no-storybook 100-flight-booking-plain