autosearch:citation-index

Citation Index — Stable URL-to-Number Map

Borrowed from STORM's StormArticle.url_to_unified_index + deepagents research_agent/prompts.py citation consolidation. Keeps citations stable across report sections and subagent outputs.

State

citation_index:
  entries:
    # URL (canonicalized) → entry
    "https://arxiv.org/abs/2401.12345":
      number: 1
      title: "First paper title"
      first_seen_section: "introduction"
      used_in_sections: ["introduction", "background", "method"]
      source_channel: "search-arxiv"
    "https://github.com/foo/bar/issues/42":
      number: 2
      title: "Issue: memory leak"
      first_seen_section: "implementation-notes"
      used_in_sections: ["implementation-notes"]
      source_channel: "search-github-issues"
  next_number: 3

URL Canonicalization Rules

Before indexing, normalize the URL:

1. Strip tracking params: utm_*, gclid, fbclid, ref=, source=.
2. Strip fragments (#section-1) unless the URL is an anchor-addressed resource (e.g. docs page).
3. Lowercase host but keep path case.
4. Collapse multiple slashes (// → /) except for the scheme.
5. Platform-specific:
   • YouTube: watch?v=ID form is canonical; strip list= playlist params.
   • arXiv: strip version suffix for de-dup (v2 → ""); keep title for reference.
   • GitHub issues: canonicalize githubissues.com/X/Y/Z to github.com/X/Y/issues/Z.

Different URL canonicalizations → different citation numbers. A URL that points to the same resource but failed canonicalization is a bug; log it and merge.

Write Path

When a subagent / section outputs evidence:

for ev in evidence:
    url = canonicalize(ev.url)
    if url not in index.entries:
        index.entries[url] = Entry(
            number=index.next_number,
            title=ev.title,
            first_seen_section=current_section,
            used_in_sections=[current_section],
            source_channel=ev.source_channel,
        )
        index.next_number += 1
    else:
        if current_section not in index.entries[url].used_in_sections:
            index.entries[url].used_in_sections.append(current_section)