infrastructure-search

Search Module

Discovery utilities for academic literature, modelled after the agent-native abstractions of Paperclip: every backend produces normalised Paper records that downstream consumers (citation export, manuscript synthesis, agent loops) can treat uniformly.

literature — Multi-source literature search

from infrastructure.search.literature import (
    Paper, SearchQuery, SearchResult, merge_papers,
    SearchBackend, LocalBackend, CrossrefBackend, ArxivBackend, PaperclipBackend,
    LiteratureClient, SearchCache,
    HttpClient, UrllibHttpClient, HttpResponse, BackendError,
)

Search across arXiv + Crossref

client = LiteratureClient([ArxivBackend(), CrossrefBackend(mailto="[email protected]")])
result = client.search(SearchQuery(text="protein language model fitness", max_results=20))
print(f"{len(result)} unique papers from {len(result.per_source_counts)} backends")
for paper in result.papers[:5]:
    print(f"  [{paper.score:.2f}] {paper.title} ({paper.year})  {paper.doi or paper.url}")

Search a local JSON corpus (offline-friendly)

backend = LocalBackend("data/curated_corpus.json")
result = LiteratureClient([backend]).search(SearchQuery(text="convex"))

Corpus format — either a list of Paper dicts or {"papers": [...]}:

[
  {
    "id": "doi:10.1126/science.1213847",
    "title": "Reproducible research in computational science",
    "authors": ["Roger D Peng"],
    "year": 2011,
    "doi": "10.1126/science.1213847",
    "venue": "Science", "venue_type": "journal"
  }
]

Search Paperclip (API key required)

import os
backend = PaperclipBackend(api_key=os.environ["PAPERCLIP_API_KEY"])
result = LiteratureClient([backend]).search(
    SearchQuery(text="GRPO hyperparameters", sources=["arxiv"], max_results=50)
)

Cache results for reproducibility

cache = SearchCache("output/search_cache", ttl_seconds=3600 * 24)
client = LiteratureClient([ArxivBackend(), CrossrefBackend()], cache=cache)

# First call hits the network and writes search_<hash>.json.
client.search(SearchQuery(text="adam optimizer"))
# Re-running the identical query is a deterministic file read.
client.search(SearchQuery(text="adam optimizer"))