scraper-mcp: Instructions file for Claude Code

Repository Guide

This file describes the current Scraper MCP implementation and verification workflow. Follow the session's AGENTS instructions for authorization, Git, and handoff requirements. Preserve unrelated edits and never read or modify secrets.

Project and commands

Scraper MCP is a Python 3.12+ application built with the MCP Python SDK's FastMCP 1.x API, Pydantic v2, requests, BeautifulSoup/lxml, and markdownify. Dependencies are managed with uv; optional extras are perplexity, playwright, and dev.

uv sync --locked --all-extras
uv run --no-sync python -m scraper_mcp
uv run --no-sync ruff check .
uv run --no-sync ruff format --check .
uv run --no-sync mypy src/ scripts/
bun install --frozen-lockfile
bun run check
uv build --no-sources

Run tests with the credential/proxy isolation shown in docs/TESTING.md. Focused pytest commands can append a file or -k filter and --no-cov. Do not install arbitrary new package versions to repair a lock mismatch; update metadata and the lock deliberately, then validate them together.

The CLI accepts positional transport, host, and port. Defaults are streamable-http, 127.0.0.1, and 8000; the alternative transport is sse. There is no stdio CLI option. Resources/prompts are opt-in through --enable-resources, --enable-prompts, or their ENABLE_* environment settings.

Code map

• server.py: FastMCP instance, transport-security defaults, tool/custom-route registration, and run_server().
• __main__.py: CLI argument parsing. __init__.py derives __version__ from installed package metadata, so install the project before importing it.
• tools/router.py: registered MCP tool signatures and delegation to services.
• tools/service.py: batch orchestration, semaphore limits, HTML transformations, sparse response metadata, and request metrics.
• models/: Pydantic response models and supported Perplexity models/providers.
• core/providers.py: default requests provider and lazy Playwright selection.
• providers/requests_provider.py: requests-backed fetches, retries, caching, proxy handling, TLS verification, and credential redaction.
• providers/playwright_provider.py: Chromium/browser-context pooling and JavaScript rendering. Browser packages are optional outside the Docker image.
• services/perplexity_service.py: direct Perplexity SDK and OpenRouter HTTP backends, model allowlisting, and normalized responses/errors.
• utils.py: BeautifulSoup/Soup Sieve selection, markdown/text conversion, link resolution, and page metadata extraction.
• cache_manager.py: public cache API, TTL policy, key generation, and statistics.
• cache_storage.py: transactional SQLite storage with strict versioned JSON.
• admin/security.py: token authentication and browser request protections.
• admin/router.py, admin/service.py: administrative HTTP handlers and volatile runtime configuration. Do not assume every reported setting changes behavior; check its consumers before changing or documenting it.
• dashboard/: authenticated HTML dashboard and template assets.
• resources/, prompts/: optional MCP read resources and prompt templates.
• scripts/check_release.py, scripts/smoke_image.py: publication guard and offline production-container verification.