scrape-structured-data

Scrape structured data from a page

chrome-agent extract finds the repeating record pattern on a page and returns it as JSON. No selectors to write, no HTML read into the model to find them.

which chrome-agent || npm install -g chrome-agent
chrome-agent goto https://news.ycombinator.com
chrome-agent --json extract --limit 50

{"count":30,"pattern":"TR","items":[
  {"title":"PGSimCity - How PostgreSQL Works","url":"https://nikolays.github.io/PGSimCity/","fields":["..."]},
  ...
]}

Check count against what the page shows. Wildly off means the page has more than one repeating pattern — scope it with --selector.

Why extract rather than read the page

Measured on the Hacker News front page. All three contain the same 30 stories; only the first hands them over as records.

approach	tokens	what you get
extract --limit 30	~1,570	all 30 stories as records with URLs
inspect (accessibility tree)	~5,650	the tree, stories mixed into the page chrome
raw HTML	~8,730	everything, including markup you'll never read

3.6x against the tree, 5.6x against raw HTML here. The margin narrows on pages that are nothing but a list: on a blog archive, ~12,500 tokens against ~16,100 for the tree. The win comes from pages where the records sit inside a lot of other markup.

Scoping, lazy loading, infinite scroll

chrome-agent extract --selector ".product-grid"     # restrict to one container
chrome-agent extract --limit 100                    # default is 10
chrome-agent extract --scroll                       # scrolls, waits for new nodes, then extracts
chrome-agent extract --a11y --scroll --limit 50     # React/SPA feeds where the DOM is noise

--a11y reads the accessibility tree instead of the DOM. Use it when a site renders into nested generated <div>s (X.com, most React apps) and plain extract returns junk.

When extract returns nothing

A page with no repeating pattern gives an empty list and a hint — the correct answer for an article or a landing page. Fall back in this order: