What is spider-mcp CLAUDE.md?

A development guide for Spider MCP, a server that connects AI assistants to Spider Cloud's web-scraping service. MCP is a standard way for AI tools to call external software; this guide also explains the TypeScript build and test commands.

Use it to work with web scraping, crawling, search, links, screenshots, and data transformation through the server, or to develop and test the TypeScript project.

It gives agents a clear map of the server and the commands needed to build, test, benchmark, and run it. This avoids guessing how the scraping tools are exposed.

How willbohn/spider-mcp configures Claude Code

Reuse

Borrowing it

Nothing to install: this file belongs to willbohn/spider-mcp. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.

Copy the file

curl -O https://raw.githubusercontent.com/willbohn/spider-mcp/main/CLAUDE.md

Clone the repo

git clone --depth 1 https://github.com/willbohn/spider-mcp

Made for: Claude Code.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/instructions/willbohn/spider-mcp/claude-md/github.svg)](https://agentmods.dev/instructions/willbohn/spider-mcp/claude-md)

Your own site

<a href="https://agentmods.dev/instructions/willbohn/spider-mcp/claude-md"><img src="https://agentmods.dev/badge/instructions/willbohn/spider-mcp/claude-md/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/instructions/willbohn/spider-mcp/claude-md"><img src="https://agentmods.dev/badge/instructions/willbohn/spider-mcp/claude-md.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 846 This file is loaded in full into every session.

When invoked 846 The same file — it is already loaded in full.

Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00846	$0.00846
Opus 5.5	$0.00338	$0.00338
Sonnet 5.5	$0.00169	$0.00169
Haiku 4.5	$0.00085	$0.00085

Measured 2mo ago against content hash be590191a13d, method: parsed. Prices are Anthropic first-party input rates as of 2026-10-07, from the pricing page.

Security

Grade A, and why

spider-mcp CLAUDE.md scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2mo ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

CLAUDE.md · 110 lines

How it starts

The opening of the file, as written. The whole thing — 110 lines — stays where its author put it; the contents beside it link to each section on GitHub.

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Build and Development Commands

# Build the TypeScript project
npm run build

# Run in development mode with auto-reload
npm run dev

# Run the compiled server
npm start

# Run test suites
npm test                    # Basic test suite
npm run test:quick         # Quick smoke tests only  
npm run test:full          # Comprehensive test suite (100+ tests)
npm run test:linkedin      # LinkedIn-specific tests (28 tests)
npm run test:category      # Run tests by category

# Run benchmark tests
npm run benchmark

# Install locally for global use
./install-local.sh         # Creates global 'spider-mcp' command

Architecture Overview

This is a Model Context Protocol (MCP) server that bridges AI assistants with the Spider Cloud web scraping API. The server implements the MCP specification using JSON-RPC 2.0 protocol.

Core Components

src/index.ts - Main server implementation (663 lines)

SpiderMCPServer class extends MCP SDK Server
Implements 6 Spider Cloud tools: scrape, crawl, search, links, screenshot, transform
HTTP client with Bearer token authentication via Axios
Retry logic with exponential backoff for transient failures
Dynamic timeout adjustment (30s default, 120s for search operations)
Comprehensive error handling with normalized error codes

Key Design Patterns

API Integration

All Spider API calls go through makeSpiderRequest() method with retry logic
Responses normalized to consistent format with success, results, count, costs, metadata
Cost tracking integrated into every response
Debug mode available via DEBUG environment variable

Parameter Validation

Zod schemas for runtime type validation
Comprehensive parameter schemas matching Spider API capabilities
Parameter normalization (e.g., waitFor → wait_for)

Error Handling Strategy

4xx errors (except 429) fail immediately - client errors
429 and 5xx errors trigger exponential backoff retry
Network/timeout errors also trigger retry
Maximum 3 retries with delays: 1s, 2s, 4s

Read the full file on GitHub · 110 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

2mo ago First seen · 110 lines · 846 tokens per session scan A be590191a13d

Subscribe to this mod's changes

spider-mcp CLAUDE.md is an instructions file published in the GitHub repository willbohn/spider-mcp (1 stars, last pushed 5d ago), licensed MIT. It adds 846 tokens to every session, about $0.0034 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.

Browser automation

spider-mcp CLAUDE.md next to 6 instructions from other repositories
Mod	Type	Stars	Last push	Licence	Scan	Tokens	Works with
spider-mcp CLAUDE.mdwillbohn/spider-mcp	Instructions file	1	5d ago	MIT	A	846	Claude Code
vscode buildNext.instructions.mdmicrosoft/vscode	Instructions file	195k	today	MIT	A	9,719	GitHub Copilot
vscode oss-third-party-notices.instructions.mdmicrosoft/vscode	Instructions file	195k	today	MIT	A	5,516	GitHub Copilot
next.js AGENTS.mdvercel/next.js	Instructions file	143k	today	MIT	A	7,461	Codex, OpenCode
langchain AGENTS.mdlangchain-ai/langchain	Instructions file	149k	today	MIT	A	4,680	Codex, OpenCode
vscode copilot-instructions.mdmicrosoft/vscode	Instructions file	195k	today	MIT	A	3,093	GitHub Copilot
vscode model-prompts.instructions.mdmicrosoft/vscode	Instructions file	195k	today	MIT	A	2,022	GitHub Copilot

spider-mcp: Instructions file for Claude Code