crawlforge-batch-automation

CrawlForge Batch & Automation

Scale up scraping and drive interactive pages. Use batch_scrape for many URLs, get_batch_results to page through async output, scrape_with_actions to interact before scraping, and generate_llms_txt to produce a site's AI policy file.

When to use

• "Scrape these 30 URLs" / "batch-scrape this list" → batch_scrape
• "Collect dozens of product / news / competitor pages" → batch_scrape (async)
• "Get the rest of the results from that batch" → get_batch_results
• "Click / type / scroll / wait before scraping" / "log in then extract" → scrape_with_actions
• "Generate an llms.txt for this site" → generate_llms_txt

batch_scrape — many URLs in parallel (cost: 5)

Sync mode (results returned immediately), good for up to ~25 URLs:

{
  "tool": "batch_scrape",
  "params": {
    "urls": ["https://a.com", "https://b.com", "https://c.com"],
    "formats": ["markdown"],
    "mode": "sync",
    "maxConcurrency": 5
  }
}

Async mode with a webhook for large batches (returns a batchId immediately):

{
  "tool": "batch_scrape",
  "params": {
    "urls": ["https://a.com", "https://b.com"],
    "formats": ["json"],
    "mode": "async",
    "webhook": { "url": "https://my-site.com/hook", "events": ["batch_completed", "batch_failed"] }
  }
}

• urls accepts plain strings OR objects {url, selectors, headers, timeout, metadata} for per-URL config. 1–50 URLs.
• formats: markdown, html, json, text.
• extractionSchema applies structured extraction to every URL.
• maxConcurrency 1–20 (default 10); delayBetweenRequests throttles.
• Sync batches over ~25 URLs trigger a confirmation prompt (elicitation) — use async for large jobs.

CLI: crawlforge batch urls.txt --format markdown --concurrency 10.

get_batch_results — page through output (cost: 1)

{ "tool": "get_batch_results", "params": { "batchId": "batch_1234567890_abc", "page": 2, "pageSize": 25 } }

Use the batchId from batch_scrape to retrieve paginated results for a completed or in-progress job. Cheap (1 credit) because the batch was already paid for. Completed jobs are also exposed as crawlforge://job/{jobId} resources. Stored batch results share the local 1-hour result store that read_result reads, with the same eviction, so page through a batch within the hour. Like batch_scrape, it takes max_inline_chars (default 40,000): a page over the limit comes back as a preview plus a result_handle for read_result.