web-scraper

Web Scraper Skill

Overview

Extract structured data from websites responsibly. Choose the right tool based on whether the page is static HTML or requires JavaScript execution, then handle pagination, rate limiting, and data cleaning.

Tool Selection

Situation	Tool
Static HTML, simple pages	requests + beautifulsoup4
JS-rendered content (SPAs, React/Vue)	playwright or selenium
Heavy scraping / crawling	scrapy
API available	Use the API - always prefer it

Step-by-Step Process

Step 1: Check for an API First

Before scraping, check:

• robots.txt at site.com/robots.txt
• The site's developer docs for a public API
• Network tab in DevTools - many "scraped" sites already have an XHR/fetch API

If an API exists, use it. It's faster, more reliable, and respectful.

Step 2: Static HTML Scraping

import requests
from bs4 import BeautifulSoup
import time

headers = {
    "User-Agent": "Mozilla/5.0 (compatible; research-bot/1.0; +https://example.com/bot)"
}

response = requests.get("https://example.com/products", headers=headers, timeout=10)
response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")

# Find elements by CSS selector
items = soup.select("div.product-card")
for item in items:
    name = item.select_one("h2.product-name").get_text(strip=True)
    price = item.select_one("span.price").get_text(strip=True)
    link = item.select_one("a")["href"]
    print(name, price, link)

Step 3: JavaScript-Rendered Pages

from playwright.sync_api import sync_playwright
import time

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/spa-page", wait_until="networkidle")

    # Wait for dynamic content
    page.wait_for_selector("div.results")

    # Extract data
    items = page.query_selector_all("div.product-card")
    for item in items:
        name = item.query_selector("h2").inner_text()
        price = item.query_selector("span.price").inner_text()
        print(name, price)

    browser.close()