research-crawl-rss

Research Crawl RSS

Argument: $ARGUMENTS (required, see Runtime Inputs)

If $ARGUMENTS is empty or missing required fields, STOP and report which fields are absent.

---

Preamble

Runtime Inputs

Parse from $ARGUMENTS:

--feed-urls=[u1,u2,...]          [JSON-style array of absolute feed URLs (RSS / Atom / RDF), non-empty; each must be an http(s) URL. Singular invocation expressed as a single-element array.]
--raw-output-dir={PATH}          [absolute path to a directory where per-feed rendered markdown files + manifest.json will be written]
--max-items-per-feed={N}         [positive integer per-feed item cap; MUST be ≥ 1, see §1.0 Step 3 + §6.0 row max-items-zero]

# Optional power params (omit when not needed):
--days={N}                       [positive integer recency window in days; default 1]
--max-body-chars={N}             [positive integer per-item body truncation cap; default 8000]

Validation rules (before §1.0):

• --feed-urls, --raw-output-dir, and --max-items-per-feed mandatory. If any missing, STOP and report which.
• --feed-urls must parse as a non-empty JSON array of strings. Else reject (feed-urls-malformed).
• Each entry must be an absolute http:// or https:// URL. First failing entry → STOP and report (feed-urls-malformed) with the offending value named.
• --raw-output-dir must be absolute. Reject relative paths (caller owns path resolution).
• --max-items-per-feed must parse as integer ≥ 1. Zero rejected per §6.0 max-items-zero.
• --days and --max-body-chars must parse as positive integers when supplied; defaults 1 and 8000.

If any validation fails, STOP and report which field failed.

Global References

RSS_INGEST_SCRIPT = scripts/rss-ingest.py    # feedparser-via-Bash leaf engine; one batched run over all feeds

Leaf does not read tool guides at runtime, the documented pitfalls are already encoded inside RSS_INGEST_SCRIPT (browser-UA default load-bearing, Accept-Encoding: identity + gzip fallback, HTML-strip before boundary-aware truncation, per-feed cap + newest-first sort-then-slice as the token governor, recency-window filtering on published/updated dates, dateless-item fallback + items_dropped_dateless surfacing, 403 gating classification cloudflare-waf vs cloudflare-js-challenge, bozo + content-type sniff + empty-body soft-failure surfacing, title-only body_sources:{none} provenance, est_tokens chars/4 budgeting heuristic). RSS fetch is free, $0 per run; the per-feed cap is a context/token governor, not a cost cap.