web-search-mcp: Instructions file for Claude Code

Web Search MCP — Project Guide

MCP server exposing web search + page-content tools. It scrapes search engine result pages through a headless browser and returns structured results.

Commands

uv sync                              # install deps (dev group includes pytest)
uv run pytest test.py -q             # run the test suite
python main.py                       # run the MCP server (stdio transport)
web-search-mcp                       # same, via the installed console script

# Run against the Obscura backend instead of Selenium/Chrome:
BROWSER_BACKEND=obscura OBSCURA_BIN="$PWD/obscura-binaries/obscura" python main.py

Architecture (main.py)

Two layers, kept deliberately decoupled:

1. Browser backends — render a URL and return (html, title):

   • BrowserBackend — abstract interface (fetch_html, close).
   • SeleniumBackend — lazy headless-Chrome driver via webdriver-manager.
   • ObscuraBackend — shells out to the Obscura CLI (obscura fetch <url> --dump html --wait-until networkidle0 --timeout N).
   • _build_backend() — selects the backend from the BROWSER_BACKEND env var (default selenium). For obscura, it resolves the binary from OBSCURA_BIN or PATH; if not found it logs a warning and falls back to Selenium.

2. WebSearcher — engine-agnostic scraping/parsing on top of a backend:

   • Multi-engine fallback (search_with_fallback): Google → DuckDuckGo → Bing. A failing engine is added to blocked_engines for the session.
   • Per-engine parsers (_parse_google/_duckduckgo/_bing_results) work on the rendered HTML with BeautifulSoup, so they are identical across backends.
   • _normalize_url unwraps redirect wrappers to the real destination: Google /url?q=, DuckDuckGo //duckduckgo.com/l/?uddg=, Bing /ck/a?...u=a1<base64url>.

MCP tools (FastMCP, decorated with @mcp.tool()): search_web, get_webpage_content, get_search_engine_status, reset_search_engines. A single lazy global WebSearcher is created via get_searcher().