scraping-at-scale

Scraping at Scale

Rule: for a large crawl, separate the catalog (what's cached) from the bodies (the data) from the action log (what the run did). Make the catalog durable on GPFS so the job is resumable, and never materialize a million loose files.

This is the heavy machinery for crawls of tens of thousands of pages or more. For the common case (WRDS, a few API pulls, credentials, a request-hash cache), use acquiring data instead — most data work never needs what's here.

Three separate stores

data/raw_html/<aa>/<key>.html   # bodies, sharded by 2-char hash prefix
data/raw_json/<aa>/<key>.json
data/derived/                   # parsed outputs
data/metadata.db                # catalog: SQLite, one row per stored artifact
data/fetch_log.jsonl            # optional: JSONL, one row per fetch attempt

Save bodies under raw, parse separately into derived — if parsing changes, re-parse without re-fetching. Then keep two records that answer different questions:

1. Catalog (metadata.db, SQLite). One canonical row per stored artifact, keyed by key: url, final_url, status, content_type, bytes, etag, last_modified, fetched_at. UPSERT on each success — a 304 revalidation just updates last_modified/fetched_at without duplicate rows. Answers "what's in the cache?"
2. Action log (fetch_log.jsonl, optional). Append-only, one row per attempt: ts, url, attempt, outcome (ok/cache_hit/retry/error), status, key, error. Answers "what did the scraper do this run?" — including failures that produced no body.

Different shapes, different formats: the catalog has one-row-per-key identity, lookup, and updates (SQLite); the log is append-only and read as a stream (JSONL, safe to multi-write under O_APPEND).

Use WAL for the catalog. SQLite's default journal (DELETE) serializes readers and writers — a DuckDB query during a scrape blocks the next upsert. WAL gives concurrent reads + serialized writes, halves per-commit fsync, and with synchronous = NORMAL is durable (a crash loses at most the last in-flight transaction, never corrupts). The helper below sets it up.