31 MCP tools that give Claude, Cursor and any MCP client the live web — scrape, crawl, search, real Google rank, change tracking, document parsing, plus an autonomous agent that researches from a plain-English prompt with no URLs. Clean Markdown and schema-validated JSON, not raw HTML. Local-Ollama extraction by default. MIT, 1,000 free credits.
This repository also configures its own agents. See what crawlforge-mcp tells them →
Skill Claude CodeCodex
Scale up scraping and drive interactive pages. Use batchscrape for many URLs, getbatchresults to page through async output, scrapewithactions to interact before scraping, and generatellmstxt to produce a site's AI policy file.
Skill Claude CodeCodex
Automates large scraping jobs and browser interactions with CrawlForge's batchscrape, getbatchresults, scrapewithactions, and generatellmstxt tools. Use when the user wants to scrape many URLs at once, batch-scrape a list of pages, collect dozens of product, news, or competitor pages, run browser actions (click, type…
Skill Claude CodeCodex
Monitors web pages for changes over time with CrawlForge's trackchanges tool. Use when the user wants to track changes to a page, watch a URL, monitor competitor pricing, detect when content updates, get notified of regulation or product-availability changes, or diff a page against a saved baseline. Workflow: create a…
Skill Claude CodeCodex
Runs multi-source web research and autonomous question-answering with CrawlForge's deepresearch, agent, searchweb, and redditsearch tools. Use when the user wants to research a topic, do a deep dive, compare competitors, gather facts with citations, answer a question from the web, search the web, or get a synthesized…
Skill Claude CodeCodex
Orientation and tool-selection guide for the CrawlForge MCP server's 31 web tools. Use when the user is getting started with CrawlForge, asks which CrawlForge tool to use, how to set up the API key, how skills or the CLI work, what a tool costs in credits, or when one tool fails and a fallback is needed. Routes…
Skill Claude CodeCodex
Bypasses bot detection with CrawlForge's stealthmode tool and looks up a country's locale settings with localization. Use when a site returns 403 or 429, CAPTCHAs, 'please enable JavaScript', or empty content, or is protected by Cloudflare, DataDome, or PerimeterX, or when the user needs a page in a specific locale…
Skill Claude CodeCodex
Extracts structured JSON and analyzes content with CrawlForge's extractstructured, extractwithllm, extractembeddedstate, scrapestructured, scrapetemplate, processdocument, analyzecontent, summarizecontent, and listollamamodels tools. Use when the user wants to extract specific fields, pull data into a JSON schema…
Skill Claude CodeCodex
Scrapes web pages and returns clean Markdown, HTML, plain text, links, or metadata using CrawlForge's scrape, fetchurl, extractcontent, extracttext, extractlinks, extractmetadata, mapsite, and crawldeep tools. Use when the user wants to scrape a URL, fetch a page, get the markdown or text of a website, extract links…
Skill Claude CodeCodex
Keeps one browser page alive across several tool calls with CrawlForge's browsersession tool. Use when the user needs to log in and then read pages behind that login, work through a multi-step flow (search, filter, paginate, fill a wizard), or look at a page before deciding what to click. The loop is: open a session…
Skill Claude CodeCodex
Builds a brand style guide or a design-token file from a live website with the branding format of CrawlForge's scrape tool. Use when the user wants a company's brand colours, fonts, logo, button styles or design tokens, asks to extract a website's style, design system or brand kit, wants a style guide made from a URL…
At most 3 mods per repository are shown here, and a mod shipped inside a plugin is left to that plugin's page — the rest are on their repository pages: