scraping-tools

Scraping Tools: Firecrawl + Exa

Fallback web scraping and search when built-in WebFetch fails (403s, bot blocks, JS-heavy pages).

When to Use

1. WebFetch returns 403/blocked - switch to Firecrawl scrape
2. Need to search the web semantically - use Exa search
3. JS-heavy pages that return empty content - use Firecrawl
4. Find similar content to a URL - use Exa findSimilar

Prerequisites

Set these environment variables (add to .env or export):

export FIRECRAWL_API_KEY="fc-..."    # Get from https://firecrawl.dev
export EXA_API_KEY="..."              # Get from https://exa.ai

Firecrawl: Scrape Any URL

Use when you need to extract content from a specific URL.

Scrape a page to markdown

curl -s -X POST 'https://api.firecrawl.dev/v2/scrape' \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H 'Content-Type: application/json' \
  -d "{\"url\": \"$URL\", \"formats\": [\"markdown\"]}" | jq -r '.data.markdown'

Scrape with specific content extraction

curl -s -X POST 'https://api.firecrawl.dev/v2/scrape' \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "'"$URL"'",
    "formats": ["markdown"],
    "onlyMainContent": true,
    "waitFor": 3000
  }' | jq -r '.data.markdown'

Search the web via Firecrawl

curl -s -X POST 'https://api.firecrawl.dev/v2/search' \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "query": "'"$QUERY"'",
    "limit": 5,
    "scrapeOptions": {"formats": ["markdown"]}
  }' | jq '.data[] | {title: .title, url: .url, content: .markdown[:500]}'

Exa: Semantic Web Search

Use when you need to find relevant content by meaning, not just keywords.

Search for content

curl -s -X POST 'https://api.exa.ai/search' \
  -H "x-api-key: $EXA_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "query": "'"$QUERY"'",
    "type": "auto",
    "numResults": 5,
    "contents": {"text": {"maxCharacters": 2000}}
  }' | jq '.results[] | {title, url, text: .text[:500]}'