surf-mcp: Instructions file for Claude Code

Surf MCP Server

Purpose: MCP server for visual browser automation via Fara.

Version: 0.5.0

---

Core Concepts

Visual Grounding

Surf uses multimodal LLMs (Fara-7B via LM Studio, or Gemini/GPT-4V) to locate UI elements by natural language description instead of brittle CSS selectors.

An AI that can see the page doesn't need to parse HTML.

Direct Fara Execution (ADR-005)

Fara returns complete tool_calls, not just coordinates. We execute what Fara decides:

{
  "name": "computer_use",
  "arguments": {
    "action": "left_click",
    "coordinate": [624, 280],
    "reasoning": "The search button is a blue element..."
  }
}

Available actions: left_click, double_click, type, scroll, key, visit_url, terminate, wait

FaraToolCall Data Model

The FaraToolCall dataclass preserves Fara's full action context:

• action: Action type (left_click, type, scroll, etc.)
• coordinate: (x, y) pixel coordinates
• text: Text to type (for type action)
• direction: Scroll direction (up/down)
• keys: Keys to press (for key action)
• confidence: Model confidence (0.0-1.0)
• reasoning: Fara's chain-of-thought explanation

Multi-Server LM Studio Support

Supports multiple LM Studio instances across different GPUs/machines:

• Server Discovery: Probes each server's /v1/models manifest
• Prefer Loaded: Prioritizes servers with Fara already loaded in VRAM
• Fallback: Sequential retry across servers on failure

Configure in .env:

LMSTUDIO_SERVERS="rtx3090=http://localhost:1234/v1,rtx8000=http://192.168.137.2:1234/v1"
FARA_MODEL_IDS="microsoft_fara-7b,fara-7b-gguf,gao-zijian/fara-7b"
FARA_MAX_FAILURES=2
FARA_PROBE_TIMEOUT=2.0

---

Installation & Running

Docker (Recommended)

docker compose up

Direct Installation (Development)

pip install -e ".[dev]"
playwright install chromium
surf-mcp  # Run the server

---

Development Directives

Testing

# Setup
pip install -e ".[dev]"
docker build --target prod -t surf-mcp .

# Run tests
pytest                           # All tests (skips unavailable)
pytest -m "not llm"              # Skip LLM tests (for CI)
pytest -m docker                 # Docker container tests
pytest -m "docker and browser"   # Docker + browser tests
pytest -m llm -v -s              # Real LLM tests (requires LM Studio)
mypy src/