web-scraping

Web Scraping Skill

Extract structured data from websites. Covers static scraping (fast, lightweight) and dynamic scraping (for JavaScript-rendered pages).

When to use what

Approach	When to use	Speed	Libraries
Static (requests + BS4)	Server-rendered HTML, APIs	Fast	requests, beautifulsoup4, lxml
Dynamic (browser)	JavaScript-rendered, SPAs	Slower	playwright, selenium

Default to static scraping. Only use browser-based scraping if the content is rendered by JavaScript.

Installation

# Static scraping
pip install requests beautifulsoup4 lxml

# Dynamic scraping (if needed)
pip install playwright && playwright install chromium

Static Scraping Patterns

Basic page fetch + parse

import requests
from bs4 import BeautifulSoup

resp = requests.get("https://example.com", timeout=10)
resp.raise_for_status()
soup = BeautifulSoup(resp.text, "lxml")

title = soup.find("h1").text
links = [a["href"] for a in soup.select("a[href]")]

Extract structured data

products = []
for card in soup.select(".product-card"):
    products.append({
        "name": card.select_one(".name").text.strip(),
        "price": card.select_one(".price").text.strip(),
        "url": card.select_one("a")["href"],
    })

Handle pagination

all_items = []
page = 1

while True:
    resp = requests.get(f"https://example.com/products?page={page}", timeout=10)
    soup = BeautifulSoup(resp.text, "lxml")

    items = soup.select(".product-card")
    if not items:
        break  # No more pages

    for item in items:
        all_items.append(item.select_one(".name").text.strip())

    page += 1
    time.sleep(1)  # Be polite

print(f"Scraped {len(all_items)} items across {page - 1} pages")

Set headers to avoid blocks

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.5",
}
resp = requests.get(url, headers=headers, timeout=10)