scraper-mcp CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

A Model Context Protocol (MCP) server for efficient web scraping. Built with Python using FastMCP, providing AI tools with standardized web scraping capabilities through four main tools: raw HTML scraping, markdown conversion, text extraction, and link extraction. All tools support both single URL and batch operations with intelligent retry logic.

Development Commands

Environment Setup

# Install dependencies (uses uv package manager)
uv pip install -e ".[dev]"

Running the Server

# Run locally with default settings
python -m scraper_mcp

# Run with specific transport and port
python -m scraper_mcp streamable-http 0.0.0.0 8000

# Run with Docker
docker-compose up -d
docker-compose logs -f
docker-compose down

Testing

# Run all tests with coverage
pytest

# Run specific test file
pytest tests/test_server.py

# Run specific test class
pytest tests/test_server.py::TestScrapeUrlTool

# Run specific test function
pytest tests/test_server.py::TestScrapeUrlTool::test_scrape_url_success

# Run with verbose output
pytest -v

# Run without coverage report
pytest --no-cov

Code Quality

# Type checking
mypy src/

# Linting
ruff check .

# Auto-fix linting issues
ruff check . --fix

# Format code
ruff format .

Architecture

Provider Pattern

The server uses an extensible provider architecture for different scraping backends:

• ScraperProvider (providers/base.py): Abstract interface defining scrape() and supports_url() methods
• RequestsProvider (providers/requests_provider.py): Default HTTP scraper using requests library with exponential backoff retry logic
• Future extensibility: Easy to add Playwright, Selenium, or Scrapy providers for JavaScript-heavy sites or specialized scraping

The get_provider() function in server.py routes URLs to appropriate providers. Currently defaults to RequestsProvider for all HTTP/HTTPS URLs.