Scrape web pages using Scrapling with anti-bot bypass (like Cloudflare Turnstile), stealth headless browsing, spiders framework, adaptive scraping, and JavaScript rendering. Use when asked to scrape, crawl, or extract data from websites; webfetch fails; the site has anti-bot protections; write Python code to…
Use Exa Agent for multi-step web research, list-building, enrichment, structured output, run continuation, and coverage validation. Exa Agent can access additional data providers: fiber, financialdatasets, similarweb, baselayer, affiliate, particle, and jinko.
Deep research powered by Exa. Use for lead generation, literature reviews, deep dives, competitive analysis, or any query where one search falls short, including phrases like 'research this', 'find everything about', 'find me all', or 'deep dive on'.
Use when extracting a page needs scripted interaction first — click, type, press a key, scroll, wait, screenshot, or run JS before capturing the DOM. Covers crawlberg interact URL --actions with the real action schema, result shape, limits, and external-CDP options.
Crawl, scrape, and convert websites to Markdown using the local crawlberg CLI and its MCP server. Use when the user wants to fetch a page, follow links across a domain, enumerate URLs, or drive a real browser. Covers installation, the subcommands (scrape, crawl, map, interact, batch-scrape, batch-crawl, download…
Use when the user wants to follow links across a domain and capture every reachable page as Markdown. Covers crawlberg crawl with depth, page caps, concurrency, rate limiting, domain scoping, robots, and output selection.
Powerful web scraping, crawling, and data extraction with stealth anti-bot bypass. Bypasses anti-bot systems (Cloudflare Turnstile, CAPTCHAs) out of the box. Use when: (1) scraping websites that block normal requests, (2) extracting structured data from web pages, (3) crawling multiple pages with concurrency, (4)…