web-scraping

Web Scraping with mq

Three tools — pick the smallest one that fits:

• mq-crawl — fetch/crawl a URL to Markdown, with JS rendering, multi-page crawling, robots.txt handling.
• mq — filter/transform Markdown or HTML (-I html) with jq-like selectors. Full selector/function reference: processing-markdown skill.
• mq's http() builtin (--allow-net) — make the request inside a query, for cases mq-crawl can't (POST, custom headers/auth).

mq-crawl sends logs/stats to stderr — redirect with 2>/dev/null when scripting.

Fetch & Extract

mq-crawl --depth 0 https://example.com 2>/dev/null                       # page as Markdown (--depth 0 = don't follow links)
mq-crawl --depth 0 -q '.h | to_text()' https://example.com 2>/dev/null   # inline query, e.g. an outline

-q output is always Markdown/plain text, regardless of -f (which only formats the stats block on stderr, not content). For JSON/table/grep, skip -q and pipe the page into mq instead:

mq-crawl --depth 0 https://example.com 2>/dev/null | mq -F json 'select(.link)'                            # structured rows
mq-crawl --depth 0 https://example.com 2>/dev/null | mq -F json '.[][]'                                     # table cells
mq-crawl --depth 0 https://example.com 2>/dev/null | mq -F grep --context 2 'select(contains("Pricing"))'   # locate text

Filter with select(...) like jq: select(.h.level <= 2), select(.code.lang == "js"), select(contains("keyword")).

JS-Rendered Pages & Multi-Page Crawls

mq-crawl --depth 0 --headless --headless-network-idle https://spa.example.com 2>/dev/null

mq-crawl https://docs.example.com --depth 2 --allowed-domains docs.example.com \
  --concurrency 4 -o ./crawled 2>/dev/null

• --headless needs local Chrome (or --webdriver-url for a remote WebDriver).
• --allowed-domains is an exact-match domain list, not a wildcard — list every subdomain you want crawled.
• -o DIR saves one Markdown file per page instead of stdout; aggregate them with mq -I null --allow-read 'collection("./crawled") | map(self, fn(page): get(page, "title");)' /dev/null (collection reads from disk, so it requires --allow-read).