imdb

IMDb (imdb.com)

Full catalog (charts, titles, names) is a Next.js app served via CloudFront with an AWS WAF challenge on bare curl requests. A real Chrome session (the real Chrome debug port) passes the challenge silently; __NEXT_DATA__ then yields a complete, typed JSON payload that's vastly richer than the DOM.

Detection

Architecture

• Next.js SSR: every chart/title page inlines a <script id="__NEXT_DATA__"> JSON blob with the full page payload already resolved.
• Chart pages (/chart/top, /chart/moviemeter, /chart/boxoffice, …) expose the list at props.pageProps.pageData.chartTitles.edges[].node.
• Each node has id (tconst), titleText, releaseYear, runtime, ratingsSummary (aggregateRating + voteCount), certificate, titleGenres, primaryImage, etc. — no DOM scraping needed.

Strategy used

• Phase 0 (curl): blocked. HTTP/2 202 + x-amzn-waf-action: challenge, zero body. Don't waste time tweaking headers — AWS WAF is JS-challenge based.
• Phase 1 (framework): skipped as a standalone fetch path — same WAF wall — but __NEXT_DATA__ is the extraction target once the page loads in a browser.
• Phase 2 (browser): puppeteer.connect({ browserURL: the real Chrome debug port }). Real Chrome clears WAF on the first navigation with no interaction. No stealth plugin, no captcha solver, no proxy.

Stealth config that works

import puppeteer from "puppeteer-core";

const browser = await puppeteer.connect({ browserURL: "http://127.0.0.1:9222" });
const page = await browser.newPage();
await page.goto("https://www.imdb.com/chart/top/", { waitUntil: "domcontentloaded", timeout: 45000 });
await page.waitForSelector("li.ipc-metadata-list-summary-item");