twitter-x-scraping

Skill: Scraping Nitter.net for Tweets

This codebase scrapes nitter.net — an open-source Twitter frontend — to collect tweets from a list or profile within a time window. The technique works without a Twitter API key.

Core Methodology

1. URL Structure

Nitter mirrors Twitter's URL layout:

2. Fetch Strategy: axios → Playwright fallback

Start with plain HTTP (axios). If nitter serves an anti-bot challenge page (detected by the string "Verifying your request" in the HTML), permanently switch to a headed Playwright/Chrome session for the rest of the run.

// challengeSolver.ts — detect challenge
const CHALLENGE_MARKER = 'Verifying your request';
export function isChallengePage(html: string): boolean {
  return html.includes(CHALLENGE_MARKER);
}

// scrapeNitterList.ts — switch modes on detection
if (isChallengePage(html)) {
  useBrowserMode = true;
  const browserHtml = await fetchWithBrowser(url, debug);
  return { html: browserHtml, url };
}

3. Browser Anti-bot Bypass

When challenge is detected, launch a real system Chrome (not headless) with automation signals suppressed:

import { chromium } from 'playwright';

const browser = await chromium.launch({
  channel: 'chrome',
  headless: false,
  args: ['--disable-blink-features=AutomationControlled'],
});

const context = await browser.newContext({ locale: 'en-US' });
await context.addInitScript(() => {
  Object.defineProperty(navigator, 'webdriver', { get: () => undefined });
});

const page = await context.newPage();
await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 30000 });

// Wait for challenge to auto-resolve
await page.waitForSelector('.timeline-item', { timeout: 30000 });
const html = await page.content();

Key: reuse a single browser/page instance across all subsequent fetches (singleton pattern) — don't re-launch per page.