crawling skills

17 tagged crawling, measured the same way as everything else here.

Browse within: csharp 11elixir 11ffi 11golang 11java 11

scrapling-official

01

D4Vinci/Scrapling

Skill Claude CodeCodex

Scrape web pages using Scrapling with anti-bot bypass (like Cloudflare Turnstile), stealth headless browsing, spiders framework, adaptive scraping, and JavaScript rendering. Use when asked to scrape, crawl, or extract data from websites; webfetch fails; the site has anti-bot protections; write Python code to…

77k 7d ago A 80 tokens original BSD-3-Clause

exa-agent

02

exa-labs/exa-mcp-server

Skill Claude CodeCodex

Use Exa Agent for multi-step web research, list-building, enrichment, structured output, run continuation, and coverage validation. Exa Agent can access additional data providers: fiber, financialdatasets, similarweb, baselayer, affiliate, particle, and jinko.

4.9k 11d ago A 58 tokens original MIT

search

03

exa-labs/exa-mcp-server

Skill Claude CodeCodex

Deep research powered by Exa. Use for lead generation, literature reviews, deep dives, competitive analysis, or any query where one search falls short, including phrases like 'research this', 'find everything about', 'find me all', or 'deep dive on'.

4.9k 11d ago A 55 tokens original MIT

xberg-io/crawlberg

Skill Claude CodeCodex

Use when extracting a page needs scripted interaction first — click, type, press a key, scroll, wait, screenshot, or run JS before capturing the DOM. Covers crawlberg interact URL --actions with the real action schema, result shape, limits, and external-CDP options.

165 3d ago A 63 tokens original MIT

crawlberg

05

xberg-io/crawlberg

Skill Claude CodeCodex

Crawl, scrape, and convert websites to Markdown using the local crawlberg CLI and its MCP server. Use when the user wants to fetch a page, follow links across a domain, enumerate URLs, or drive a real browser. Covers installation, the subcommands (scrape, crawl, map, interact, batch-scrape, batch-crawl, download…

165 3d ago A 107 tokens original MIT

crawling-a-site

06

xberg-io/crawlberg

Skill Claude CodeCodex

Use when the user wants to follow links across a domain and capture every reachable page as Markdown. Covers crawlberg crawl with depth, page caps, concurrency, rate limiting, domain scoping, robots, and output selection.

165 3d ago A 52 tokens original MIT

abx-dl

07

ArchiveBox/abx-dl

Skill Claude CodeCodex

Use this when working on the standalone ArchiveBox downloader CLI, extractor orchestration, plugin execution, config, and output verification.

141 4d ago C 30 tokens original MIT

LeoYeAI/openclaw-ultra-scraping

Skill Claude CodeCodex

Powerful web scraping, crawling, and data extraction with stealth anti-bot bypass. Bypasses anti-bot systems (Cloudflare Turnstile, CAPTCHAs) out of the box. Use when: (1) scraping websites that block normal requests, (2) extracting structured data from web pages, (3) crawling multiple pages with concurrency, (4)…

22 2mo ago A 180 tokens original MIT