better-fetch

BetterFetch

Fetch and convert web pages to clean Markdown. Supports single-page fetch and multi-page crawl, with automatic failover between providers.

Provider Chain

1. Primary — markdown.new: https://markdown.new/{URL}
2. Fallback — Jina Reader: https://r.jina.ai/{URL}
3. Last resort: Fetch the original URL directly.

Failover triggers: HTTP 403/429, empty response, bot-detection pages, timeout, or any non-200 error from the current provider. When falling back, briefly note which provider failed and why.

Jina Reader Authentication (optional)

Jina Reader works without authentication but has stricter rate limits. To increase quota, pass a bearer token via the Authorization header:

Authorization: Bearer jina_<your_token>

Get a free API key at https://jina.ai/reader/.

Input

The user provides URLs and options, optionally followed by a prompt or question.

Mode Detection

• Single-page mode (default): No flags, or --method=browser|ai|auto
• Crawl mode: When --crawl flag is present, or the user explicitly says "crawl" / "crawl the entire site"

Single-Page Instructions

1. Parse the input: extract URL(s) and any user prompt.
2. Fire all requests concurrently. When multiple URLs are provided, launch all fetches in parallel — do not wait for one URL to finish before starting the next. Each URL independently follows the provider chain:
   • markdown.new: Fetch https://markdown.new/{URL} with the instruction "Return the full markdown content as-is. Do not summarize."
   • Jina Reader (on failure): Fetch https://r.jina.ai/{URL} with the same instruction. Jina Reader returns Markdown directly.
   • Direct fetch (on failure): Fetch the original URL and note the fallback.
3. Collect all results. If the user provided a prompt/question, process the combined markdown to answer it. Otherwise, present each page's content in order.

Crawl Instructions

Crawl is only available via markdown.new. If the crawl API itself is blocked, inform the user and suggest single-page mode with Jina Reader as an alternative.