scrapling-fetch-mcp: Instructions file for Claude Code

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What it does

scrapling-fetch-mcp is an MCP server that helps AI assistants fetch web content from bot-protected sites. It exposes two tools (s_fetch_page, s_fetch_pattern) via FastMCP over stdio, using Scrapling (patchright + playwright + curl-cffi) to bypass anti-automation measures and returning HTML or Markdown.

It also ships as a Claude Code skill with a /s-fetch slash command that invokes scrapling directly via Bash. The skill requires the MCP server to be installed first.

Commands

# Install dependencies
uv sync

# Install browser binaries (required once; large download)
uvx --from scrapling-fetch-mcp scrapling install

# Run MCP server
uv run scrapling-fetch-mcp

# Build
uv build

No test suite exists in this project.

Skill structure

skills/s-fetch/SKILL.md              # /s-fetch — fetches URLs via scrapling
skills/s-fetch/references/install.md # one-time install steps, loaded only on miss

Install: clone the repo and copy skills/s-fetch into ~/.claude/skills/ (see README for commands).

/s-fetch uses $(uv tool dir)/scrapling-fetch-mcp/bin/python3 -. The first time it runs on a machine without the tool, SKILL.md's Setup section directs Claude to references/install.md, which is read on demand so it doesn't sit in context once the tool is present.

Architecture

Request flow: mcp.py (FastMCP tool definitions) → _fetcher.py (content pipeline) → _scrapling.py (Scrapling dispatch) → returns html_content.

_scrapling.py — sole integration point with Scrapling. Maps the three fetch modes:

• basic — AsyncFetcher.get with stealthy headers (fastest, ~1-2s)
• stealth — StealthyFetcher.async_fetch headless+network_idle (~3-8s)
• max-stealth — same but with WebRTC blocked and resources/images not blocked (10s+)

Scrapling import is deferred inside browse_url with stdout redirected to /dev/null to suppress Scrapling's noisy startup output.