crawlforge-web-scraping

CrawlForge Web Scraping

Fetch and clean web content with the CrawlForge MCP server. This skill covers single-page scraping (one URL → Markdown / HTML / text / links / metadata), site discovery (sitemaps and URL maps), and whole-site crawling.

When to use

• "Scrape this URL" / "get the markdown of this page" / "read this article" → scrape
• "Just give me the raw HTML / JSON / headers" → fetch_url
• "Give me the clean article text, no nav or ads" → extract_content
• "Get the plain text / markdown body" → extract_text
• "List every link on this page" → extract_links
• "Get the title / meta / Open Graph / SEO tags" → extract_metadata
• "Map all the URLs on this site" / "generate a sitemap" → map_site
• "Crawl the whole docs site" / "index every page" → crawl_deep
• "Read more of that truncated result" → read_result

Tool selection (fastest path first)

1. scrape is the default for single pages. One fetch returns many formats at once — no N-request fan-out. Ask for exactly the formats you need.
2. Use the single-purpose basic tools (fetch_url, extract_text, extract_links, extract_metadata) when you only want one cheap thing.
3. extract_content when you specifically want Readability-cleaned article body (strips ads, nav, footers) — better than extract_text for articles.
4. map_site before a crawl to estimate scope / find section URLs.
5. crawl_deep to walk an entire site and (optionally) extract content.

If a page returns 403/429, a CAPTCHA, or empty "enable JavaScript" content, switch to the crawlforge-stealth-browsing skill (stealth_mode). When the site is known to block, use scrape with escalate: true instead of two calls — still never stealth_mode first.

scrape — unified multi-format (cost: 2)

Get markdown + links + metadata in a single call:

{
  "tool": "scrape",
  "params": {
    "url": "https://example.com/article",
    "formats": ["markdown", "links", "metadata"],
    "onlyMainContent": true
  }
}