mcp-web-docs: Cursor rule for Cursor

Cursor Rules for MCP Web Docs

This is a TypeScript MCP (Model Context Protocol) server project for crawling and indexing web documentation.

Quick Reference

For comprehensive guidelines, see AGENTS.md in the project root.

Project Overview

MCP Web Docs is a self-hosted MCP server that:

• Crawls and indexes documentation from any website
• Provides hybrid search (full-text + semantic/vector)
• Supports authentication for protected documentation sites
• Uses local embeddings (FastEmbed) for privacy

Tech Stack

• Runtime: Node.js >= 24, npm >= 11
• Language: TypeScript (ES2022, NodeNext modules)
• Testing: Vitest
• Storage: SQLite (metadata) + LanceDB (vectors)

Essential Rules

Code Style

• TypeScript with strict mode enabled
• Always use .js extensions in imports (ESM requirement)
• Follow existing patterns in the codebase
• Run npm run lint and npm run prettier before committing

Testing

• Tests are co-located: *.test.ts next to source files
• Use Vitest globals: describe, it, expect, vi
• Mock embeddings: createMockEmbeddings() from src/__mocks__/embeddings.ts
• Use vi.hoisted() for mocks configured before imports

Critical Constraints

• Never log to stdout - MCP servers use stdio for JSON-RPC
• Use logger from src/util/logger.ts (writes to stderr)
• Validate inputs with Zod schemas from src/util/security.ts
• Use validatePublicUrl() for SSRF protection

Key Commands

npm test              # Run tests
npm run build         # Build TypeScript
npm run lint          # Run ESLint
npm run prettier      # Format code
npm run test:coverage # Test with coverage
npm run test:types    # Type check

File Structure

src/
├── index.ts          # CLI bootstrap and signal handling
├── server.ts         # MCP server and tool handlers
├── crawler/          # Web crawling (Playwright/Crawlee)
├── processor/        # Content processing
├── storage/          # SQLite + LanceDB
├── embeddings/       # FastEmbed vectors
├── indexing/         # Workflow, status tracking, queue management
└── util/             # Logger, security, helpers