web-source-fetching

Web Source Fetching Strategies

This document contains rules and strategies for fetching authoritative content from various web sources. It is a living document — add new entries as you discover what works for each website.

---

Core Principles

1. Never rely on internal world knowledge for technical content — every claim must trace to a downloaded source
2. Source of truth hierarchy: LaTeX/source code > HTML > PDF > summaries
3. Download first, read later — save content locally before processing
4. Prefer raw sources — GitHub raw files, arXiv LaTeX, not rendered HTML
5. search_web is for discovery only — use it to identify sources, not to extract content

---

Anti-Patterns to Avoid

❌ Don't Do This	✅ Do This Instead
Use search_web for content	Use it only for source discovery
Fabricate URLs	Only use URLs you've actually fetched
Regenerate content from memory	Quote/cite downloaded sources
Use browser for static pages	Use curl or read_url_content
Link to external URLs for images	Download images locally
Conclude a source is incomplete because grep found no matches	List section headings first (grep '^##'), then read relevant sections. A grep false negative is not evidence of missing content.

---

Extraction Completeness Verification (MANDATORY for Web Sources)

Readability is necessary but not sufficient. A web extraction can produce a readable, substantial .md file that is missing entire sections due to a soft paywall, JavaScript rendering failure, or extraction timeout. This is especially common with Substack (free preview + paywalled body) and Medium.

After extracting any web article, verify structural completeness:

1. List section headings: grep '^##\|^###\|^####' content.md — the output should show a logical article structure (introduction, body sections, conclusion).
2. Check for paywall markers: grep -i 'upgrade to paid\|subscribe to continue\|for paid subscribers\|unlock this post' content.md — if any of these strings appear, the extraction hit a paywall boundary. Re-extract with --profile for the relevant site (e.g., --profile substack).
3. Check the ending: Read the last 20 lines of the file. Does the article end with a conclusion/summary, or does it cut off abruptly with a subscription prompt? An abrupt ending signals truncation.
4. Compare against expected content: If you know the article discusses topics X, Y, and Z (from web search summaries or the TEXTBOOK-PLAN), verify that the section headings include all three topics. If topic Z is missing from the headings, the extraction may be incomplete.