scrape-batch

scrape-batch

Fan out extract(action="batch") over a URL list wet already knows, then report per-URL outcomes honestly. The batch path applies per-domain politeness (2 concurrent and 1 request/second per domain, 6 fetches in flight overall) and returns whatever succeeded even when some URLs fail.

Use this skill when:

• The user supplies a list of URLs to read in full.
• A previous search returned hits and the user wants the bodies, not the snippets.
• A crawl or extract(action="map") produced a URL set to pull down.

Do NOT use this skill when:

• There is one URL, or a handful from one domain -- call extract(action="extract", urls=[...]), which is cached and cheaper.
• The URLs are not known yet and the goal is an answer, not the pages -- use the research-topic skill (extract(action="agent")).
• The target is a whole site rather than a list -- use extract(action="crawl") or extract(action="map").
• The user wants images or video from the pages -- use media(action="list") then media(action="download").

Steps

1. Collect and de-duplicate the URL list. Drop duplicates and fragment-only variants (#section) -- each one costs a full fetch. Report the final count to the user before spending it.

2. Split into chunks of at most 50. The cap is hard: 51 URLs returns {"error": "Error: Maximum 50 URLs per batch (got 51)"} and nothing is fetched -- the call is refused, not truncated.

3. Split further when one domain dominates. Politeness is per domain, so 40 URLs on a single host serialise to roughly one per second while 40 URLs across 20 hosts run near the global limit. A whole tool call is capped at 120 seconds (TOOL_TIMEOUT), and hitting that ceiling returns only {"error": "... timed out after 120s ..."} -- the already-fetched pages are lost with it. Keep single-domain batches near 15-20 URLs.

4. Call one chunk at a time, waiting for each to return:

   extract(action="batch", urls=[...], format="markdown")
   

   format accepts markdown (default), text or html. Pass stealth=true only after a normal attempt returns near-empty content for a protected site; it escalates to the heavier fetch strategies.