stackoverflow

Stack Overflow (stackoverflow.com)

Classic server-rendered HTML in front of Cloudflare. A single cold curl gets HTML 200, but every subsequent anonymous request is 403'd by Cloudflare's bot heuristics (the __cf_bm / _cfuvid cookies issued on the first hit need a real browser TLS fingerprint to stay valid). Real Chrome via 127.0.0.1:9222 sidesteps this entirely. Question data is inline in the HTML — no API or JSON blob needed.

Detection

Signal	Value
CDN	Cloudflare (server: cloudflare, cf-ray)
Framework	ASP.NET / Razor (server-rendered HTML, no SPA)
Anti-bot	Cloudflare bot scoring — 403s node/curl after 1 hit
Auth	Not required for question listings
robots.txt	Allow: /questions/ on major crawlers

Architecture

• /questions/tagged/<tag> is fully SSR — every question is a .s-post-summary block in the raw HTML.
• Pagination is a plain ?tab=newest&page=N&pagesize=50 query param. pagesize accepts 15 / 30 / 50.
• There is a "Stack Exchange API" at api.stackexchange.com — useful but heavily rate-limited (300 req/day unauthenticated, with quota key required). For one-off tag scrapes, scraping the HTML is simpler.

Strategy used

• Phase 0 (curl): first request returns 200 with full HTML. Second request: 403. Cloudflare is stateful.
• Phase 1: no framework JSON blob — data is straight HTML. Skip.
• Phase 2: puppeteer.connect({ browserURL: the real Chrome debug port }). First navigation succeeds, and so does every subsequent one — the real Chrome profile keeps the Cloudflare cookies rotating correctly.

Stealth config that works

import puppeteer from "puppeteer-core";

const browser = await puppeteer.connect({
  browserURL: "http://127.0.0.1:9222",
});
const page = await browser.newPage();
await page.goto(url, { waitUntil: "domcontentloaded", timeout: 30000 });
await page.waitForSelector(".s-post-summary");
const html = await page.content();
// ...
await page.close();
browser.disconnect(); // never .close() — it would kill the user's Chrome