26 tagged crawler, measured the same way as everything else here.
MCP server Claude CodeCodexCursor +2
Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy and effortless as it should be! Runs locally from the scrapling Python package.
MCP server Claude CodeCodexCursor +2
MCP server for CRW web scraper — scrape, crawl, map, extract, search, and PDF-parse tools for AI agents. Runs locally from the crw-mcp npm package.
MCP server Claude CodeCodexCursor +2
Scrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents. Remote server at fastcrw.com.
MCP server Claude CodeCodexCursor +2
MCP server "webreaper" as configured in alex-on-ai/WebReaper. Launched with WebReaper.Mcp.
MCP server Claude CodeCodexCursor +2
Search the web and extract clean, readable text from webpages. Process multiple URLs at once to sp… Remote server at server.smithery.ai.
MCP server Claude CodeCodexCursor +2
MCP server "crawlie", hosted remotely at crawlie.app, as configured in spronta/crawlie.
MCP server Claude CodeCodexCursor +2
MCP server "deepspider" as configured in ma-pony/deepspider. Runs src/mcp/server.js with node.
MCP server Claude CodeCodexCursor +2
Crawl and analyse websites for SEO errors using Crawlee with SQLite storage. Runs locally from the @houtini/seo-crawler-mcp npm package. Needs 2 environment variables to run.
MCP server Claude CodeCodexCursor +2
Technical SEO in Claude: GSC + crawl + DataForSEO - audit, market sizing, content recon. Runs locally from the @houtini/seo-audit-console npm package. Needs 9 environment variables to run.
serkan-ozal/driflyte-mcp-server
MCP server Claude CodeCodexCursor +2
Discover available topics and explore up-to-date, topic-tagged web content. Search to surface the… Remote server at server.smithery.ai.
serkan-ozal/driflyte-mcp-server
MCP server Claude CodeCodexCursor +2
Driflyte MCP server which lets AI assistants query topic-specific knowledge from web and GitHub. Runs locally from the @driflyte/mcp-server npm package.
MCP server Claude CodeCodexCursor +2
Web scraping, crawling, deep research & autonomous extraction — 29 MCP tools, clean Markdown/JSON. Runs locally from the crawlforge-mcp-server npm package. Needs 3 environment variables to run.
MCP server Claude CodeCodexCursor +2
Universal web scraper with LLM-ready markdown, RAG chunking, PDF/DOCX support. Runs locally from the thecrawler npm package.
kopachlager/prerenderbuddy-plugins
MCP server Claude CodeCodexCursor +2
Local MCP tools for public crawler diagnostics and optional Prerender Buddy workspace evidence. Runs locally from the @prerenderbuddy/mcp npm package.
MCP server Claude CodeCodexCursor +2
MCP server "bxc-native-mcp" as configured in aphrody-code/bxc. Runs ${CLAUDE_PROJECT_ROOT}/dist/standalone/bxc-mcp with node. Needs 1 environment variable to run.
kopachlager/prerenderbuddy-mcp
MCP server Claude CodeCodexCursor
Local MCP tools for crawler-readability and discovery-file diagnostics. Runs locally from the @prerenderbuddy/mcp npm package.
MCP server Claude CodeCodexCursor
MCP server "crawmcp" as configured in bilginerdem/craw-mcp. Runs locally from the crawmcp npm package.
MCP server Claude CodeCodexCursor
MCP server for the Synoppy web-data API — read, screenshot, crawl, map, search, extract, classify, enrich, and images as tools for any MCP client. Runs locally from the @synoppy/mcp npm package.
MCP server Claude CodeCodexCursor
Native Codex SEO CLI for technical audits, performance, Search Console, baselines, historical trends, and CI quality gates. Runs locally from the codex-seo npm package.
maxaeo/maxaeo-ai-visibility-mcp
MCP server Claude CodeCodexCursor
Local-first MCP server for AI visibility, GEO/AEO, llms.txt, and AI crawler readiness audits. Runs locally from the maxaeo-ai-visibility-mcp npm package.
MCP server Claude CodeCodexCursor
Eval-backed tool discovery for AI agents on the auxiliar.ai web-access gateway. recommendtools picks the best search, scraping, browser-automation or voice provider for a job from measured benchmarks (quality, latency, cost, error rate), returning routes. Runs locally from the auxiliar-mcp npm package.
MCP server Claude CodeCodexCursor
Eval-backed discovery for the auxiliar.ai gateway — the best web-access provider per job, measured. Runs locally from the auxiliar-mcp npm package.
MCP server Claude CodeCodexCursor
Model Context Protocol server for web crawling powered by Crawl4AI. Runs locally from the mcp-crawl4ai Python package.
MCP server Claude CodeCodexCursor +2
Give any LLM agent the power to read any website — even the ones that block bots. An MCP server that returns clean Markdown and handles blocks automatically via residential proxies. Runs locally from the hatfetch npm package.