collecting-sources

Collecting Sources

Phase 1 of two-phase research: discover, evaluate, and store sources with metadata. Synthesis (Phase 2) and fact-checking (Phase 3) are separate phases — this skill ends at handoff.

Everything lands in the task workspace:

work/<task-id>/
├── source_index.md      # human-readable catalogue
└── sources/
    ├── papers/          # academic papers (.txt from CORE/Unpaywall, .pdf from arXiv)
    ├── web/             # web pages (Markdown)
    ├── blogs/           # blog posts (Markdown)
    └── docs/            # documentation (Markdown/HTML)

Every source file has a .meta.yaml companion conforming to schema v1.0 — fields, missing-data conventions, and a complete example in references/source-metadata-schema.md.

1 — Setup

• mkdir -p work/<task-id>/sources/{papers,web,blogs,docs}
• Source IDs are sequential and zero-padded: src_001, src_002, …
• If source_index.md exists from a previous collection, load it for duplicate detection.

2 — Discovery

Detect the domain from the research question:

• Academic: "paper", "study", "peer-reviewed", "journal"; biomedical (clinical, drug, patient), CS/physics (algorithm, neural network, quantum), social sciences (policy, governance, economics); academic URLs given (arxiv.org, doi.org, pubmed.gov)
• General web: "tutorial", "guide", "how-to", documentation focus, no academic terminology

Academic research — discovery finds papers and extracts identifiers (DOI, arXiv ID, PubMed ID, exact title) for the retrieval chain; do not fetch full text during discovery. Run 3–5 targeted WebSearch queries mixing plain and site-specific forms:

• Biomedical: site:pubmed.ncbi.nlm.nih.gov [topic]
• CS/physics: site:arxiv.org [topic]
• Any domain: site:scholar.google.com [topic], [topic] research paper peer-reviewed

Collect candidate metadata: title, authors, publication date, venue, abstract excerpt, identifiers.

General web research — run 3–5 targeted WebSearch queries with alternative phrasings; target authoritative sources (official docs, expert blogs, reputable sites); use site-specific searches when the site is known (site:docs.python.org async). Collect URLs, titles, authors, dates.