web-scraping

Web Scraping

Extract structured data from websites, handling both static HTML and JavaScript-rendered content (React, Next.js, Vue, etc.).

When to Use

• User asks to "scrape", "extract", or "get data from" a website
• Target site uses client-side rendering (SPA frameworks)
• Need to interact with dynamic content (infinite scroll, lazy loading)
• API endpoints are not available or documented

Approach Selection

1. Static HTML (curl + parsing)

Use when: Site serves complete HTML without JavaScript rendering.

curl -sL 'https://example.com' | grep -oP 'pattern'
# or with jq for JSON APIs
curl -s 'https://api.example.com/data' | jq '.items[]'

Pros: Fast, lightweight, no dependencies
Cons: Fails on JS-rendered content

2. Headless Browser (Puppeteer/Playwright)

Use when: Content is rendered client-side (React, Next.js, Vue, Angular).

Node.js + Puppeteer (recommended for WSL2/containers):

const puppeteer = require('puppeteer');

const browser = await puppeteer.launch({
  headless: true,
  args: ['--no-sandbox', '--disable-setuid-sandbox']  // Required in WSL2/containers
});

const page = await browser.newPage();
await page.goto('https://example.com', {
  waitUntil: 'networkidle2',
  timeout: 60000
});

// Wait for dynamic content
await new Promise(resolve => setTimeout(resolve, 3000));

// Extract text
const content = await page.evaluate(() => document.body.innerText);

// Extract structured data
const data = await page.evaluate(() => {
  return Array.from(document.querySelectorAll('.item')).map(el => ({
    title: el.querySelector('.title')?.innerText,
    value: el.querySelector('.value')?.innerText
  }));
});

await browser.close();

Pros: Handles all JS rendering, can interact with page
Cons: Slower, heavier resource usage

3. API Inspection (DevTools Network tab)

Use when: Site loads data via XHR/fetch calls.

1. Open browser DevTools → Network tab
2. Filter by XHR/Fetch
3. Find API endpoint
4. Replicate with curl/fetch