github

GitHub (github.com)

GitHub's public HTML pages are fully server-rendered with no JS gating and no bot protection on unauthenticated GETs. A one-shot fetch + cheerio parse is all you need. Only reach for the REST API (api.github.com) when you hit the 60 req/hr unauthenticated limit or need structured fields (topics, license, default branch) not present on the HTML.

Detection

Signal	Value
CDN	GitHub edge (Varnish/Fastly)
Framework	Rails + Turbo, React islands
Anti-bot	None on public pages
Auth	Optional — anon works
Rate limit (HTML)	Generous, IP-based, soft
Rate limit (REST)	60/hr unauth, 5000/hr with token

Architecture

• /trending is a normal Rails view. Repo cards are <article class="Box-row"> inside <div data-hpc>.
• Note: the trending list size varies — historically 25, but currently GitHub often returns only ~10 cards. Don't hardcode the count.
• Repo pages (/{owner}/{name}) have data in multiple places: HTML meta tags, <react-app> embedded JSON payloads, and DOM nodes. For simple fields (stars, forks, description), raw DOM is easiest.
• Sitemaps at /sitemap.xml are huge — prefer targeted URL lists.

Strategy used

Phase 0 curl → done. Data is in raw HTML, no protection headers, no JS required. Skip browser entirely.

Extraction (trending)

import { load } from 'cheerio';

const res = await fetch('https://github.com/trending', {
  headers: { 'User-Agent': 'Mozilla/5.0' },
});
const $ = load(await res.text());

$('article.Box-row').each((_, el) => {
  const $el = $(el);
  const fullName = $el.find('h2 a').attr('href').replace(/^\//, '').replace(/\s+/g, '');
  const description = $el.find('p').text().trim() || null;
  const language = $el.find('[itemprop="programmingLanguage"]').text().trim() || null;
  const stars = parseInt($el.find('a[href$="/stargazers"]').text().trim().replace(/,/g, '')) || 0;
  const forks = parseInt($el.find('a[href$="/forks"]').text().trim().replace(/,/g, '')) || 0;
  const starsToday = $el.find('span.d-inline-block.float-sm-right').text().trim().replace(/\s+/g, ' ');
  // ...
});