scrape

Web Scrape to Markdown

Fetch a URL and convert it to clean, LLM-ready Markdown. Supports multiple backends with automatic fallback.

Usage

/devkit:scrape https://example.com
/devkit:scrape https://example.com --json
/devkit:scrape https://example.com https://other.com
/devkit:scrape https://example.com --backend playwright

Arguments

• One or more URLs to scrape
• --json — wrap output as Markdown-in-JSON: { "url": "...", "title": "...", "markdown": "..." }
• --backend <name> — force a specific backend (see below). Default: auto-detect.

Backends (in priority order)

1. Jina Reader — prepend https://r.jina.ai/ to the URL. Returns Markdown by default. Use if JINA_API_KEY is set (higher rate limits) or anonymously (~20 RPM). Best for articles and docs.
2. Playwright — use if npx playwright --version succeeds (optional dep, install with npx playwright install chromium). Best for JS-heavy SPAs and sites that block headless scrapers. Free and local — no API keys.
3. Camoufox — patched Firefox with leak-fixed JA3/TLS fingerprints. Use if python -m camoufox version succeeds (install: pip install -U camoufox[geoip] && python -m camoufox fetch). Free and local. Use when Playwright returns a challenge page (Cloudflare "Just a moment...", DataDome, PerimeterX). Auto-triggered when Playwright response body matches challenge signatures (cf-chl-bypass, __cf_chl_, _Incapsula_Resource).
4. Scweet (X/Twitter only) — specialized scraper for x.com / twitter.com URLs. Use if SCWEET_COOKIES_PATH is set (points at a JSON cookie file exported from a logged-in browser). Auto-routed when host is x.com or twitter.com — other backends don't work reliably on X post-API lockdown.
5. Firecrawl — use if FIRECRAWL_API_KEY is set. Paid API. Last-resort anti-bot bypass when stealth browsers still get blocked.
6. WebFetch fallback — use Claude's built-in WebFetch tool. No API key needed, but returns raw content (less clean).