agentmods.dev

spider-mcp CLAUDE.md

spider-mcp CLAUDE.md is an instructions file for coding agents from willbohn/spider-mcp. It costs 846 tokens per session, scanned A, original, MIT.

A development guide for Spider MCP, a server that connects AI assistants to Spider Cloud's web-scraping service. MCP is a standard way for AI tools to call external software; this guide also explains the TypeScript build and test commands.

In plain words
What is it for?
Use it to work with web scraping, crawling, search, links, screenshots, and data transformation through the server, or to develop and test the TypeScript project.
Why use it?
It gives agents a clear map of the server and the commands needed to build, test, benchmark, and run it. This avoids guessing how the scraping tools are exposed.

Instructions file

not rated 1 12mo ago A scan 846 tokens original MIT
View the source file willbohn/spider-mcp
Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

agentmods
npx agentmods add instructions/willbohn/spider-mcp/claude-md
Clone the repo
git clone --depth 1 https://github.com/willbohn/spider-mcp

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

agentmods badge for spider-mcp CLAUDE.md

README.md
[![agentmods](https://agentmods.dev/badge/instructions/willbohn/spider-mcp/claude-md.svg)](https://agentmods.dev/instructions/willbohn/spider-mcp/claude-md)
Your own site
<a href="https://agentmods.dev/instructions/willbohn/spider-mcp/claude-md"><img src="https://agentmods.dev/badge/instructions/willbohn/spider-mcp/claude-md.svg" alt="Measured on agentmods" height="20"></a>
Per session 846 This file is loaded in full into every session.
When invoked 846 The same file — it is already loaded in full.
Security scan A 0 findings. Scan, not verified.
Origin original No closer match found in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5 $0.00846 $0.00846
Opus 5 $0.00423 $0.00423
Sonnet 5 $0.00169 $0.00169
Haiku 4.5 $0.00085 $0.00085

Measured 3d ago against content hash be590191a13d, method: parsed. Prices are Anthropic first-party input rates as of 2026-08-30, from the pricing page.

Security

Grade A, and why

spider-mcp CLAUDE.md scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 3d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

CLAUDE.md · 110 lines

How it starts

The opening of the file, as written. The whole thing — 110 lines — stays where its author put it; the contents beside it link to each section on GitHub.

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Build and Development Commands

# Build the TypeScript project
npm run build

# Run in development mode with auto-reload
npm run dev

# Run the compiled server
npm start

# Run test suites
npm test                    # Basic test suite
npm run test:quick         # Quick smoke tests only  
npm run test:full          # Comprehensive test suite (100+ tests)
npm run test:linkedin      # LinkedIn-specific tests (28 tests)
npm run test:category      # Run tests by category

# Run benchmark tests
npm run benchmark

# Install locally for global use
./install-local.sh         # Creates global 'spider-mcp' command

Architecture Overview

This is a Model Context Protocol (MCP) server that bridges AI assistants with the Spider Cloud web scraping API. The server implements the MCP specification using JSON-RPC 2.0 protocol.

Core Components

src/index.ts - Main server implementation (663 lines)

  • SpiderMCPServer class extends MCP SDK Server
  • Implements 6 Spider Cloud tools: scrape, crawl, search, links, screenshot, transform
  • HTTP client with Bearer token authentication via Axios
  • Retry logic with exponential backoff for transient failures
  • Dynamic timeout adjustment (30s default, 120s for search operations)
  • Comprehensive error handling with normalized error codes

Key Design Patterns

API Integration

  • All Spider API calls go through makeSpiderRequest() method with retry logic
  • Responses normalized to consistent format with success, results, count, costs, metadata
  • Cost tracking integrated into every response
  • Debug mode available via DEBUG environment variable

Parameter Validation

  • Zod schemas for runtime type validation
  • Comprehensive parameter schemas matching Spider API capabilities
  • Parameter normalization (e.g., waitFor → wait_for)

Error Handling Strategy

  • 4xx errors (except 429) fail immediately - client errors
  • 429 and 5xx errors trigger exponential backoff retry
  • Network/timeout errors also trigger retry
  • Maximum 3 retries with delays: 1s, 2s, 4s

Read the full file on GitHub · 110 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 3d ago First seen · 110 lines · 846 tokens per session scan A be590191a13d

Subscribe to this mod's changes

spider-mcp CLAUDE.md is an instructions file published in the GitHub repository willbohn/spider-mcp (1 stars, last pushed 12mo ago), licensed MIT. It adds 846 tokens to every session, about $0.0042 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.

Browser automation
Related

Other instructions, from other repositories

Browser4 CLAUDE.md

Claude Code instructions for platonai/Browser4, covering browser4 — project context for claude, architecture, key dispatch chain (cli → browser), batch commands and e2e test structure.

platonai/Browser4 · 1,205 tokens

fix-quera AGENTS.md

Instructions for AlirezaKeshavarz83/fix-quera, covering agent guidance, project shape, visual style, quera page data findings and compatibility findings.

AlirezaKeshavarz83/fix-quera · 1,338 tokens

fast-browser CLAUDE.md

Claude Code instructions for m4ttstack/fast-browser, covering fast browser plugin, where a change belongs, fork branch: use fast-browser-runtime, releasing a new runtime and re-pinning this repo: use the script.

m4ttstack/fast-browser · 1,446 tokens

sendblue-browser-use AGENTS.md

Instructions for sendblue-api/sendblue-browser-use, covering agents.md — sendblue-browser-use, what this repo is, setup, common commands and health (no auth).

sendblue-api/sendblue-browser-use · 1,064 tokens

browser_oxide AGENTS.md

AGENTS.md instructions for yfedoseev/browser_oxide, covering browseroxide — conventions for ai coding assistants, build & test, architecture, key conventions and where to read more.

yfedoseev/browser_oxide · 1,202 tokens

shantycrawl-mcp AGENTS.md

AGENTS.md instructions for schlemperdev/shantycrawl-mcp, covering shantycrawl mcp -- dynamic firecrawl mcp server, purpose, ownership, local contracts and work guidance.

schlemperdev/shantycrawl-mcp · 1,820 tokens