web-research

Web Research

Fetch web pages, index them locally, and read selectively using Julie's tools. This replaces dumping entire web pages into context (which wastes thousands of tokens) with a workflow that indexes the content and lets you pull out just the sections you need.

Prerequisites

browser39 must be installed. Check with:

which browser39

If missing, tell the user to download the latest binary release from browser39 releases and add it to their PATH.

Workflow

Step 0 (Optional): Discover URLs from a Topic

If the user gave a research topic rather than a specific URL, find candidate pages first:

browser39 search "your research topic" --output json > /tmp/b39-search.json

Parse the links[] array and pick the URLs worth fetching. Skip this step when the user already provided URLs.

Step 1: Fetch and Save

Determine the target file path from the URL. The directory structure mirrors the URL:

docs/web/
  docs.rs/axum/latest.md
  developer.mozilla.org/Web/API/Fetch_API.md
  github.com/tokio-rs/tokio.md

Fetch the page and save directly to the target file. Never print full page content to stdout.

# Fetch (rm ensures no stale results — --output appends, not overwrites)
echo '{"id":"1","action":"fetch","v":1,"seq":1,"url":"THE_URL","options":{"selector":"article","strip_nav":true,"include_links":true}}' > /tmp/b39-cmd.jsonl
rm -f /tmp/b39-out.jsonl
browser39 batch /tmp/b39-cmd.jsonl --output /tmp/b39-out.jsonl

# Extract markdown directly to file (no stdout leak)
mkdir -p docs/web/TARGET_DOMAIN/TARGET_PATH_DIR
python3 -c "
import sys,json
with open('/tmp/b39-out.jsonl') as f:
    for line in f:
        d=json.loads(line)
        md = d.get('markdown','')
        if md:
            with open('docs/web/TARGET_DOMAIN/TARGET_PATH.md','w') as out:
                out.write(md)
            print(f'Saved {len(md)} chars to docs/web/TARGET_DOMAIN/TARGET_PATH.md')
"