web-intelligence

web-intelligence

Use this when the agent needs external web evidence and the question is not satisfied by one generic browser fetch.

Goal

Route web work to the right provider with a stable contract:

• exa-search for discovery, ranking, and source finding
• firecrawl-extract for clean content extraction, crawl expansion, sitemap mapping, and markdown capture
• brightdata-collection for hostile, dynamic, geo-sensitive, or SERP-specific collection

Routing Rules

1. Start with exa-search when the task begins with a question, entity, or topic rather than known URLs.
2. Use firecrawl-extract when URLs are already known and the task needs readable page content.
3. Use brightdata-collection when the target requires browser rendering, geo targeting, anti-bot resilience, or SERP capture.
4. Prefer a staged pipeline over direct heavy collection:
   • discovery via Exa
   • extraction via Firecrawl
   • hard-target recovery via Bright Data
5. Use Bright Data sparingly. It is the most operationally expensive and should not be the default fetch path.

Normalized Output Contract

Return provider results in this shape whenever practical:

{
  "query": "string",
  "goal": "source_discovery|page_extraction|site_crawl|serp_capture",
  "items": [
    {
      "title": "string",
      "url": "string",
      "snippet": "string",
      "content_markdown": "string",
      "source_provider": "exa|firecrawl|brightdata",
      "retrieved_at": "ISO-8601",
      "confidence": 0.0
    }
  ],
  "notes": ["string"]
}

Decision Table

Task shape	Preferred provider	Reason
Find best sources on a topic	Exa	semantic search and ranking
Extract article text from known URL	Firecrawl	clean markdown and crawl tools
Crawl a docs site or sitemap	Firecrawl	purpose-built crawl/map endpoints
Capture Google results in a country	Bright Data	browser and geo collection
Extract blocked JS-heavy page	Bright Data	rendering and anti-bot resilience
Fill citation gaps from discovered URLs	Exa then Firecrawl	separate finding from extraction