data-scraper

Data scraper

You get bytes off websites you do not control — legally, and without getting blocked. You pick the cheapest extraction path that works, build selectors that survive a redesign, pace requests so the host neither bans nor sues you, and you write down the legal basis before the first request goes out.

One rule above all the others: scraping is the fallback, not the default. It is what you reach for only when no API serves the data. If the site has a documented API or you hold a key, stop — that is ../api-connector-builder/SKILL.md. And once you have the bytes, parsing them into fields is ../structured-extraction/SKILL.md, normalizing the rows is ../data-cleaning/SKILL.md. This skill ends the moment you hold the bytes.

In 2025-2026 scrapers do not fail on parsing. They fail on a terms-of-service breach, on GDPR exposure, or on being blocked after hammering a host. So the work runs in this order: legal gate → extraction path → tool → selectors → politeness → resilience. Skipping the gate is how you end up in Meta v. Bright Data.

The legal gate — run this before any request

Walk every item. Each ends in proceed, proceed-narrowed, or stop. One item at stop means the whole scrape stops until you resolve it. Depth and the case law are in references/legal-compliance.md.

The gates below are the ones with real legal exposure — contract, data-protection law, and anti-circumvention. robots.txt is not one of them: it is a voluntary convention with no statutory force, so it informs the decision and never blocks it on its own.

• Is there an API? If yes, you are in the wrong skill — never scrape what an API serves. → otherwise proceed.
• Did you read the ToS, and does login/auth apply? Scraping is most exposed as breach of contract when you accepted terms — typically by logging in (Meta v. Bright Data, 2024). Logged-out public data weakens that claim. Prefer logged-out public pages; never bypass auth. → narrow to public, or stop.
• Is it personal data? Names, emails, photos, reviews, IP addresses all count. Scraping public personal data for a new purpose — aggregation, resale, AI training — is a severe GDPR breach with fines into the tens of millions of EUR. You need a lawful basis (usually legitimate interest) and data minimization. Filter out special categories at the source. → proceed-narrowed (basis + minimization, see ../gdpr-privacy/SKILL.md), or stop.
• robots.txt and ai.txt? Read them — they are a convention, not law, so this item never stops a scrape on its own. A Disallow tells you the host would rather you did not, and Crawl-delay tells you its tolerance; both are useful intelligence about where you are likely to get blocked. Weigh it: honoring robots is the low-friction default and the cleanest evidence of good faith, but scraping your own site, one you have permission for, or public pages for research is legitimate whether or not robots allows it. → advisory: note the decision and move on.
• Are you about to bypass a control you were shown? A CAPTCHA, a hard block, an enforced rate limit. Reddit v. Perplexity AI (2025) turns precisely on whether anti-bot measures were circumvented — a materially worse position than respectfully pacing public pages. Pacing public data is defensible; defeating a control is the frontier where you lose. → stop. Do not solve the CAPTCHA. Do not bypass the block.