hackernews

Hacker News (news.ycombinator.com)

HN is the easiest scrape target on the internet. The entire front page is a single server-rendered HTML table with stable, decade-old class names. No cookies, no JS, no rate limits at normal pace. If you need more than a few pages, use the official Firebase API instead of scraping.

Detection

Signal	Value
Server	nginx (Arc / Common Lisp app behind it)
CDN	None
Framework	Server-rendered HTML tables, no SPA
Anti-bot	None
Auth	Not required for reading
robots.txt	Allows /, disallows /x?, /r?, /vote?, etc. Crawl-delay: 30

Architecture

Every listing page (/news, /newest, /front, /ask, /show, /jobs, ?p=N) is a single <table id="hnmain"> with 30 stories. Each story is two <tr> rows:

1. tr.athing.submission — id, rank, title link, site domain
2. the next <tr> — subtext row with points, user, age, comments

The "More" link at the bottom (a.morelink[rel=next]) gives the next page URL, e.g. ?p=2.

Strategy used

• Phase 0 (curl): HTTP 200, all data in raw HTML. Gate A passed — skipped browser entirely.
• Phase 1/2 not needed.

Stealth config that works

None needed. Plain fetch works. A generic UA is polite:

headers: { 'user-agent': 'Mozilla/5.0 (compatible; torch-scraper/1.0)' }

Respect Crawl-delay: 30 from robots.txt if you hit many pages.

Extraction

import * as cheerio from 'cheerio';

const html = await (await fetch('https://news.ycombinator.com')).text();
const $ = cheerio.load(html);
const items = [];

$('tr.athing.submission').each((_, el) => {
  const $row = $(el);
  const id = $row.attr('id');
  const rank = parseInt($row.find('.rank').text(), 10);
  const $a = $row.find('.titleline > a').first();
  const title = $a.text().trim();
  const url = $a.attr('href');
  const site = $row.find('.sitestr').text().trim() || null;

  const $sub = $row.next().find('.subtext .subline');
  const points = parseInt($sub.find('.score').text(), 10) || 0;
  const user = $sub.find('.hnuser').text().trim() || null;
  const time = ($sub.find('.age').attr('title') || '').split(' ')[0] || null;
  const age = $sub.find('.age a').text().trim();
  const cmatch = $sub.find('a').last().text().match(/(\d+)\s*comment/);
  const comments = cmatch ? parseInt(cmatch[1], 10) : 0;

  items.push({ rank, id, title, url, site, points, user, comments, age, time,
               hn_url: `https://news.ycombinator.com/item?id=${id}` });
});