What is webcrawl-mcp CLAUDE.md?

Repository instructions for choosing web-scraping tools: fetching pages, searching, discovering URLs, and crawling multiple pages.

Use them when implementing or operating page scraping, site search, URL mapping, or multi-page crawls in this project.

They specify which webcrawl operation to try first and when to fall back to Firecrawl for JavaScript-heavy or failed requests.

How andyliszewski/webcrawl-mcp configures Claude Code

Reuse

Borrowing it

Nothing to install: this file belongs to andyliszewski/webcrawl-mcp. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.

Copy the file

curl -O https://raw.githubusercontent.com/andyliszewski/webcrawl-mcp/main/CLAUDE.md

Clone the repo

git clone --depth 1 https://github.com/andyliszewski/webcrawl-mcp

Made for: Claude Code.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/instructions/andyliszewski/webcrawl-mcp/claude-md/github.svg)](https://agentmods.dev/instructions/andyliszewski/webcrawl-mcp/claude-md)

Your own site

<a href="https://agentmods.dev/instructions/andyliszewski/webcrawl-mcp/claude-md"><img src="https://agentmods.dev/badge/instructions/andyliszewski/webcrawl-mcp/claude-md/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/instructions/andyliszewski/webcrawl-mcp/claude-md"><img src="https://agentmods.dev/badge/instructions/andyliszewski/webcrawl-mcp/claude-md.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 204 This file is loaded in full into every session.

When invoked 204 The same file — it is already loaded in full.

Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00204	$0.00204
Opus 5.5	$0.00082	$0.00082
Sonnet 5.5	$0.00041	$0.00041
Haiku 4.5	$0.00020	$0.00020

Measured 2mo ago against content hash 5648eda7b256, method: parsed. Prices are Anthropic first-party input rates as of 2026-10-07, from the pricing page.

Security

Grade A, and why

webcrawl-mcp CLAUDE.md scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2mo ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

CLAUDE.md · 23 lines

What it actually says

Project Instructions

Web Scraping Tool Preferences

When fetching web content, prefer the webcrawl MCP tools over Firecrawl:

Task	Use	Fallback
Single page	`webcrawl_scrape`	`mcp__firecrawl__firecrawl_scrape`
Search	`webcrawl_search`	`mcp__firecrawl__firecrawl_search`
Discover URLs	`webcrawl_map`	`mcp__firecrawl__firecrawl_map`
Multi-page crawl	`webcrawl_crawl`	`mcp__firecrawl__firecrawl_crawl`

When to fall back to Firecrawl:

webcrawl_scrape returns empty or very short content (JS-heavy site)
Site requires JavaScript rendering
webcrawl tool fails or times out

Why webcrawl first:

Free (no API costs)
Faster for static content
No rate limits

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

2mo ago First seen · 23 lines · 204 tokens per session scan A 5648eda7b256

Subscribe to this mod's changes

webcrawl-mcp CLAUDE.md is an instructions file published in the GitHub repository andyliszewski/webcrawl-mcp (16 stars, last pushed 2mo ago), licensed MIT. It adds 204 tokens to every session, about $0.0008 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.

Browser automation

webcrawl-mcp CLAUDE.md next to 6 instructions from other repositories
Mod	Type	Stars	Last push	Licence	Scan	Tokens	Works with
webcrawl-mcp CLAUDE.mdandyliszewski/webcrawl-mcp	Instructions file	16	2mo ago	MIT	A	204	Claude Code
vscode buildNext.instructions.mdmicrosoft/vscode	Instructions file	195k	today	MIT	A	9,719	GitHub Copilot
next.js AGENTS.mdvercel/next.js	Instructions file	143k	today	MIT	A	7,461	Codex, OpenCode
vscode oss-third-party-notices.instructions.mdmicrosoft/vscode	Instructions file	195k	today	MIT	A	5,685	GitHub Copilot
langchain AGENTS.mdlangchain-ai/langchain	Instructions file	149k	today	MIT	A	4,680	Codex, OpenCode
vscode copilot-instructions.mdmicrosoft/vscode	Instructions file	195k	today	MIT	A	3,093	GitHub Copilot
vscode model-prompts.instructions.mdmicrosoft/vscode	Instructions file	195k	today	MIT	A	2,022	GitHub Copilot

webcrawl-mcp: Instructions file for Claude Code