searchcans-rag-source-curator

SearchCans RAG Source Curator

Build a deliberately bounded source manifest rather than crawling the web. The Skill searches Google or Bing, chooses diverse candidate domains, fetches only the selected pages with Reader, accepts explicit PDF/Office file URLs, and can retain screenshot URLs for visual review.

Create the source set

Start with a concrete question, market, source budget, and the minimum number of successfully extracted sources needed before the pack is considered ready. Add direct documents only with --file-url; do not infer file URLs from a page.

python scripts/rag_source_curator.py "What is the current SERP API market?" \
  --engine google --country us --language en \
  --source-budget 4 --min-claim-ready 2 --include-content \
  --out rag-sources.json

Use --query for a small query matrix. Use --screenshot 1 or 2 only when visual review is required. --include-content intentionally makes the output larger; omit it when you need a metadata manifest before a second retrieval step.

Apply the evidence quality gate

The output marks a source claim_ready only when the Reader/File Extraction response has content. Its evidence_gate.status is passed only when the requested minimum is met.

Never treat a SERP snippet, an unread URL, an empty response, or a screenshot as sufficient evidence for a consequential answer. Every source begins with authority_assessment: unassessed; assign authority, recency, permissions, and organization-specific trust labels through a separate human or policy-based review.

Use the source-manifest guide when deciding whether the output may enter a RAG index.

Control cost and concurrency

By default, --account-mode auto uses one safe Account API pre-flight, estimates search plus the entire Reader/File source budget, caps source count when possible, and limits workers to reported Parallel Lanes. Use enforce to stop an insufficient job, warn to record without capping, or off only when an account check is inappropriate.