create-scraper

Your goal is to explore a company's careers page and write a permanent, reliable Scraper subclass that other code can use going forward. The argument is either:

1. the name of a company, e.g. /create-scraper netflix
2. a careers page URL, e.g. /create-scraper https://explore.jobs.netflix.net/careers, or
3. a json file containing a list of company names, e.g. /create-scraper .data/companies.json (in this case, you'll want to go through each company and determine if any existing scrapers work for it, or if a new one needs to be made)

Step 1a - Identify the url for the careers page

If the careers page has been provided as the argument, move to the next step; otherwise search "[company-name] careers" via the WebSearch tool and navigate until you find the careers page.

Step 1b — Identify the company name

Derive the class name and filename from the company's domain or brand name. E.g.:

• explore.jobs.netflix.net → NetflixScraper → src/analysis/scraping/NetflixScraper.ts
• jobs.stripe.com → StripeScraper → src/analysis/scraping/StripeScraper.ts

Step 2 — Navigate and capture network traffic

The goal is to find a JSON API the page calls internally — these are far more reliable than DOM scraping.

1. Navigate to the URL with browser_navigate
2. Immediately inject a network monitor via browser_evaluate:

window.__reqs = [];
const _f = window.fetch.bind(window);
window.fetch = function (input, init) {
  const url =
    typeof input === 'string'
      ? input
      : input instanceof URL
        ? input.href
        : input.url;
  window.__reqs.push({ url, method: (init?.method || 'GET').toUpperCase() });
  return _f(input, init);
};
const _open = XMLHttpRequest.prototype.open;
XMLHttpRequest.prototype.open = function (method, url) {
  window.__reqs.push({ url: String(url), method: method.toUpperCase() });
  return _open.apply(this, arguments);
};

1. Take a browser_snapshot to understand the page structure
2. Interact with the page to trigger data loads: scroll to the bottom, wait a moment, then retrieve what was captured: