web-scraping-javascript-sites

Web Scraping JavaScript/SPA Sites

The Core Insight

For SPA/TypeScript sites: try urllib FIRST before launching a browser.

TypeScript sites (Next.js, Nuxt, React, Vue) send full HTML on first request — the content is there. Browser tools show empty snapshots because they capture before JS finishes rendering. A raw HTTP fetch often gets all the data you need.

Decision Tree

Is the site an SPA (React/Next/Nuxt/Vue)?
│
├── YES → Try urllib/requests FIRST
│         └─ Got content? → DONE
│         └─ Content empty? → Use Playwright
│
└── NO (static HTML) → Use urllib/requests directly

Layer 1: Raw HTML Fetch (Fastest)

import urllib.request
import re

url = "https://example.com"
req = urllib.request.Request(url, headers={
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36'
})
response = urllib.request.urlopen(req, timeout=10)
html = response.read().decode('utf-8')

# Extract content
# Option A: regex search
prices = re.findall(r'€[\d,]+', html)

# Option B: find section context
section = re.search(r'id="pricing".{0,5000}', html, re.DOTALL | re.IGNORECASE)
if section:
    content = re.sub(r'<[^>]+>', ' ', section.group())
    content = re.sub(r'\s+', ' ', content).strip()

# Option C: parse HTML properly
from html.parser import HTMLParser
class ContentParser(HTMLParser):
    def __init__(self):
        super().__init__()
        self.in_target = False
        self.text = []
    def handle_starttag(self, tag, attrs):
        if tag in ('h1','h2','h3','p', 'li'):
            self.in_target = True
    def handle_data(self, data):
        if self.in_target:
            self.text.append(data.strip())
    def handle_endtag(self, tag):
        if tag in ('h1','h2','h3','p', 'li'):
            self.in_target = False

Why urllib Often Works for "Lazy" Sites

Many TypeScript/React sites pre-render HTML on the server (SSR) or send full content in initial HTML, but use JS to make it look animated/lazy. The data is in the first response — you just need to parse it.