docs

CrawlForge Skill Reference

Auto-generated from src/skills/agent-skills/*/SKILL.md. To regenerate, run npm run skills:gen (calls concatenateSkills() from src/skills/installer.js).

This file is the canonical capabilities reference for AI agents using CrawlForge MCP tools.

---

CrawlForge Batch & Automation

Scale up scraping and drive interactive pages. Use batch_scrape for many URLs, get_batch_results to page through async output, scrape_with_actions to interact before scraping, and generate_llms_txt to produce a site's AI policy file.

When to use

• "Scrape these 30 URLs" / "batch-scrape this list" → batch_scrape
• "Collect dozens of product / news / competitor pages" → batch_scrape (async)
• "Get the rest of the results from that batch" → get_batch_results
• "Click / type / scroll / wait before scraping" / "log in then extract" → scrape_with_actions
• "Generate an llms.txt for this site" → generate_llms_txt

batch_scrape — many URLs in parallel (cost: 5)

Sync mode (results returned immediately), good for up to ~25 URLs:

{
  "tool": "batch_scrape",
  "params": {
    "urls": ["https://a.com", "https://b.com", "https://c.com"],
    "formats": ["markdown"],
    "mode": "sync",
    "maxConcurrency": 5
  }
}

Async mode with a webhook for large batches (returns a batchId immediately):

{
  "tool": "batch_scrape",
  "params": {
    "urls": ["https://a.com", "https://b.com"],
    "formats": ["json"],
    "mode": "async",
    "webhook": { "url": "https://my-site.com/hook", "events": ["batch_completed", "batch_failed"] }
  }
}

• urls accepts plain strings OR objects {url, selectors, headers, timeout, metadata} for per-URL config. 1–50 URLs.
• formats: markdown, html, json, text.
• extractionSchema applies structured extraction to every URL.
• maxConcurrency 1–20 (default 10); delayBetweenRequests throttles.
• Sync batches over ~25 URLs trigger a confirmation prompt (elicitation) — use async for large jobs.