120-utilities

Command-Line Utilities & Documentation Ingestion Tools

Guiding principle

Choose the lightest tool that reliably produces the content you need in a machine-consumable form.

• If the page is static HTML: prefer curl plus parsing (jq for JSON, HTML to text conversion, or a lightweight extractor).
• If the page is JavaScript-rendered or requires interaction: use a headless browser (Playwright).
• If Playwright is blocked or too heavy: try a doc-extraction proxy/cache (for example https://context7.com/) when it supports the target site.
• If you need diagrams understood (not OCR): use a screenshot (Playwright) and a vision-capable model (VLM).
• If you need text inside images: use OCR (Tesseract) as a supplement.
• If you are reading official documentation at scale: prefer a documentation-aware retrieval system (RAG) over raw scraping.

What "done" looks like for an agent

The output you hand to the LLM should be:

• Clean text (minimal navigation noise)
• Source-attributed (URL, section headings)
• Chunked (so the LLM does not get one huge blob)
• Rate-limited and cacheable
• Deterministic when possible (same input URL gives similar chunks)

Tool selection matrix

1) Static pages, APIs, feeds

Use these when content is already present in HTML or JSON without JS:

• curl for fetching
• jq for JSON shaping
• ripgrep for local searching
• lynx -dump for fast text extraction

Examples:

# Fetch HTML
curl -fsSL "https://acme.com/page" -o page.html

# Fetch JSON and shape it
curl -fsSL "https://api.acme.com/v1/items" | jq '.items[] | {id, name, updated_at}'

# Extract readable text quickly
lynx -dump -nolist "https://acme.com/page" > page.txt

When to stop here:

• If the text is good enough for the agent to answer questions
• If you do not need diagrams interpreted
• If the page is not JS-rendered

2) JS-heavy sites, auth flows, dynamic docs, robust extraction

Use Playwright when curl or lynx fails to capture the real content.