brightdata-collection

brightdata-collection

Use this only when simpler search or extraction paths are insufficient.

Best Uses

• geo-specific SERP capture
• JavaScript-heavy targets
• anti-bot-protected sites
• browser-rendered collection
• difficult competitor pages that normal HTTP scraping cannot access reliably

Required Env

• BRIGHTDATA_API_KEY

Optional:

• BRIGHTDATA_ZONE
• BRIGHTDATA_COUNTRY

Invocation Threshold

Bright Data is not the default. Use it when one of these is true:

• the target requires rendering
• the target is blocked to normal fetch tools
• the task requires region-specific result pages
• the task explicitly needs search-result capture or browser automation

Collection Rules

• state why Bright Data is required before use
• keep requested geography explicit
• minimize pages collected to reduce cost
• save the exact target pattern when a route succeeds

Example Task Shapes

• capture US desktop SERP for a keyword
• fetch a React app page that returns empty shells to non-browser clients
• collect pricing pages from a site with aggressive bot defenses

Avoid

• general source discovery that Exa can handle
• straightforward content extraction from static pages that Firecrawl can handle

Residential Proxy via Playwright

For browser automation that requires a residential IP (e.g. LinkedIn, which blocks datacenter IPs at the Cloudflare layer), route Playwright through BrightData's residential proxy. Set the proxy at browser.newContext() level — not chromium.launch(). context.request (Playwright's APIRequestContext) does not inherit the launch-level proxy and returns 407.

const proxy = {
  server:   'http://brd.superproxy.io:22225',
  username: 'brd-customer-hl_XXXXX-zone-ZONE-session-STICKY_ID',
  password: 'ZONE_PASSWORD',
};

const browser = await chromium.launch({ args: ['--no-sandbox'] });
const context = await browser.newContext({ proxy });  // ← correct level
// Now both page.goto() and context.request.get() route through the proxy