mining-pubmed-literature

Mining PubMed & PMC literature (NCBI E-utilities)

Search PubMed (citations/abstracts) and PMC (full text) programmatically with NCBI E-utilities — the stable HTTP interface to Entrez. The core pattern is two steps: ESearch returns matching record IDs (PMIDs), then EFetch (or ESummary) downloads the records. The Entrez History server (usehistory=y) lets you chain the two without re-sending thousands of IDs.

E-utilities are public. No key is required, but a free API key raises your limit from 3 to 10 requests/second and is strongly recommended for batch work.

When to use

• OpenMed extracted a diagnosis, drug, or gene and you want supporting literature.
• You need abstracts to summarize or to assemble a corpus for biomedical NER.
• You want MeSH-anchored, reproducible searches (date ranges, article types).

For ClinicalTrials.gov use searching-clinicaltrials; this skill is for the published literature.

Quick start (real E-utilities calls)

Base URL: https://eutils.ncbi.nlm.nih.gov/entrez/eutils/. JSON for ESearch/ ESummary via retmode=json; EFetch returns text or XML (no JSON for PubMed).

import requests, time

BASE = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils"
API_KEY = None   # set to your free NCBI key to get 10 req/s instead of 3

def _params(**kw):
    if API_KEY:
        kw["api_key"] = API_KEY
    return kw

def esearch(term: str, retmax: int = 50) -> dict:
    """Find PMIDs; usehistory=y stores them on the Entrez History server."""
    r = requests.get(f"{BASE}/esearch.fcgi", params=_params(
        db="pubmed", term=term, retmax=retmax,
        usehistory="y", retmode="json"), timeout=30)
    r.raise_for_status()
    res = r.json()["esearchresult"]
    return {"count": int(res["count"]), "ids": res["idlist"],
            "webenv": res["webenv"], "query_key": res["querykey"]}

def efetch_abstracts(webenv: str, query_key: str, retmax: int = 50) -> str:
    """Pull abstracts by reference to the stored result set (no ID list needed)."""
    r = requests.get(f"{BASE}/efetch.fcgi", params=_params(
        db="pubmed", WebEnv=webenv, query_key=query_key,
        retmax=retmax, rettype="abstract", retmode="text"), timeout=60)
    r.raise_for_status()
    return r.text

hits = esearch('("type 2 diabetes"[MeSH]) AND metformin AND 2023:2025[pdat]')
print(hits["count"], "papers")
abstracts = efetch_abstracts(hits["webenv"], hits["query_key"])