scientific-metadata-extraction

Scientific Metadata Extraction

Overview

Patterns for extracting and normalizing metadata from scientific data repositories (GEO, SRA/ENA, PRIDE, BioStudies, PubMed) and publication sources. Covers per-sample identifier derivation, author name normalization, assay type determination, species verification, file format normalization, and a 4-tier gap-fill strategy for resolving missing annotation fields.

Installation

pip install biopython httpx

Authentication

from Bio import Entrez
import os

Entrez.email = "[email protected]"
if os.environ.get("NCBI_API_KEY"):
    Entrez.api_key = os.environ["NCBI_API_KEY"]   # raises limit from 3 to 10 req/s

Core Concepts

• Per-sample metadata: Annotation fields that describe biological properties (tissue, genotype, sex, condition) vary between samples and must be fetched from per-sample records — never copied from a single study-level value.
• Repository submitter ≠ PI: Repository submitter fields reflect whoever deposited the files, often a postdoc or research engineer. Investigator/PI fields should come from the PubMed AuthorList.
• Run accession ≠ biological ID: SRR/ERR/DRR accessions identify sequencing runs, not specimens. Use BioSample IDs or sample titles as specimen identifiers.
• Assay type from library metadata: Publication titles describe biology, not technology. Always verify assay type from library_strategy/library_source in repository records.

Common Operations

Fetch Study Leads from PubMed

Always derive investigator names from the PubMed AuthorList — first author + last/corresponding author in "Firstname Lastname" format.

from Bio import Entrez

def fetch_study_leads(pmid: str) -> list[str]:
    """Return [first_author, last_author] as 'Firstname Lastname' strings."""
    handle = Entrez.efetch(db="pubmed", id=str(pmid), rettype="xml", retmode="xml")
    recs = Entrez.read(handle)
    authors = recs["PubmedArticle"][0]["MedlineCitation"]["Article"]["AuthorList"]

    def fmt(auth) -> str:
        fore = str(auth.get("ForeName", ""))
        last = str(auth.get("LastName", ""))
        return f"{fore} {last}".strip()

    leads = [fmt(authors[0]), fmt(authors[-1])]
    return list(dict.fromkeys(leads))   # deduplicate if single-author paper

# Reformat GEO-style "Lastname FI" contributors
import re
def normalize_author_name(name: str) -> str:
    """'Smith JP' → 'J Smith', 'Doe, Jane' → 'Jane Doe'"""
    if "," in name:
        parts = [p.strip() for p in name.split(",", 1)]
        return f"{parts[1]} {parts[0]}"
    tokens = name.split()
    if len(tokens) == 2 and len(tokens[1]) <= 3 and tokens[1].isupper():
        return f"{tokens[1][0]} {tokens[0]}"
    return name