web-crawling

Web Crawling

Use this skill for general multi-page crawling when the user wants more than a one-page fetch but does not yet need a highly specialised workflow.

First choose the right crawl path

Prefer crawl4ai when:

• the user wants markdown from many pages
• the output is for AI/RAG ingestion
• documentation export is the main goal
• you want a bounded deep crawl with good markdown output

Prefer katana when:

• the main need is URL discovery or site mapping
• you want to enumerate a site's surface before extraction
• you need JS-aware endpoint discovery

Prefer scrapegraph-ai when:

• the main need is structured extraction rather than broad crawling
• you only need a small set of pages turned into JSON-like facts

Use this web-crawling skill directly when:

• you need a bounded general crawl
• a simple recursive content pass is enough
• you are combining native fetches, small scripts, and selective link following

Default operating pattern

1. Start from the seed URL.
2. Use the agent's native web fetch / simple HTTP path first for cooperative pages.
3. Extract internal links and keep scope bounded.
4. Save output to a file, not the chat transcript.
5. Increase depth only after the first pass looks good.

Escalation contract

When the crawl path hits blocking or rendering problems:

1. Try the native path first

   • native web_fetch
   • requests/BeautifulSoup
   • or the chosen crawl tool directly

2. Escalate to camofox if the problem is:

   • Cloudflare / anti-bot challenge
   • blank or JS-only page content
   • CAPTCHA / verify-you-are-human flow
   • a browser-rendering dependency that breaks the crawl

3. Escalate to residential-proxy only if all of these are true:

   • camofox was actually tried and still failed, or the blocked target is a narrow public HTTP endpoint where a browser step is not sufficient
   • the blocked URLs are high-value
   • the user explicitly approves paid proxy use
   • the proxy scope stays narrow and targeted