fetchaller-mcp: Instructions file for Claude Code

fetchaller-mcp

MCP server for fetching any URL without domain restrictions. Full Reddit support. Built-in web search.

Architecture

fetchaller owns content processing + MCP tools. wafer (~/code/wafer) owns HTTP transport + anti-detection. fetchaller NEVER does bot solving, impersonation, or cookie management — if a site blocks requests, fix it in wafer. See docs/architecture.md for full details.

Escalate to wafer only for active blocking — bot detection, a WAF challenge needing a solve, TLS rejection, clearance cookies, rate limiting. Anything that is merely finding the right request — reading JS bundles, guessing an endpoint, working out a required field, decoding a payload shape, telling one JSON blob from another — is content analysis and must be built here. Before writing anything down for wafer, ask: is this request being refused, or do I just not know its shape yet? Only the first is wafer's. src/fetchaller/discovery/ exists for the second — see docs/spa-discovery.md. This boundary was settled empirically: the discovery capability was built inside wafer, validated against seven boards, and discarded because none of them needed a challenge solved (wafer-feedback.md is the record).

Web search

Expose only search(query, provider, page=1) for web search. provider is required in MCP calls: prefer google, or choose ddg, brave, or all. Google failures fall back to DDG then Brave and must disclose the failure and actual result source. Google uses the Google Docs research-search JSON endpoint with the verified pool in src/fetchaller/search/google_documents.json, randomized healthy IDs, shared six-second pacing and document cooldowns. No Opera Mini, ordinary Google SERP, account cookies or browser. Do not rotate IDs after HTTP 429; honor Google-wide backoff and Retry-After. Search engines match Reddit's default ten-request-per-minute budget per provider with at least six-second spacing. DDG/Brave add jitter, serialize the full HTTP exchange, disable hidden wafer retries, and enable wafer's own six-second limiter. DDG's page 1 is asked once more, through the same gate, when it answers its "No results found" block: it sometimes sends that for an ordinary query. Two such answers are DDG's failure, never an empty result, so Google's fallback goes on to Brave: on 2026-11-05 it sent that block to every query for minutes, from macOS and the image, to a GET and its form's POST. A 429 holds that provider for 60 seconds; a 403 (or DDG's blocking 202) holds it for 300 seconds, extended by longer Retry-After. Do not bypass the provider gate for all or Google fallback. Brave receives the server's existing guarded, audited BrowserSolver (default solve budget 120s); its wafer request is bounded to 150s including solving, with a 180s provider budget for queueing, pacing and parsing. Google and DDG remain browser-free. Only wafer handles Brave PoW/blind tokens. Fetchaller 3.8.11 requires published wafer 0.7.13 or later (Linux and Windows platformVersion as Chrome reports it; 0.7.12 added Kasada-signed calls for costco.com and Chrome 157; 0.7.11 carried add_cookie() cookies into the browser; 0.7.10 added Reddit's refused reads and GraphQL on the app route; 0.7.9 first carried the Brave handler), with its PyPI artifact locked. Unresolved challenges, including HTTP 200 gates, are errors and start a provider cooldown. Never create an unrestricted search browser. Wafer 0.7.9 includes d01c811, fixing the accepted-reload deadlock and preserving gate evidence during fast spare-token recovery. Native headed/headless checks passed on macOS, and guarded MCP checks passed on macOS and Linux. Token cookies are consumed by HTTP reads; spare tokens stay session-local. challenge_absent=False can mean new work or spare reuse, so do not infer fresh PoW or restart persistence from that flag alone. The independent release approval and evidence are in docs/brave-search-investigation.md and ../wafer/todo-brave-search.md. See docs/google-search-fetchaller-handoff.md and the Search Architecture section in docs/architecture.md.