bio-batch-downloads

Version Compatibility

Reference examples tested with: BioPython 1.83+, Entrez Direct 21.0+

Before using code patterns, verify installed versions match. If versions differ:

• Python: pip show <package> then help(module.function) to check signatures

If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

Batch Downloads

"Download thousands of sequences from NCBI" → Search NCBI with history server, then batch-fetch results with rate limiting and retry logic.

• Python: Entrez.esearch(), Entrez.efetch() with usehistory='y' (BioPython)

Download large numbers of records from NCBI efficiently using the history server, batching, and proper rate limiting.

Required Setup

from Bio import Entrez
import time

Entrez.email = '[email protected]'  # Required by NCBI
Entrez.api_key = 'your_api_key'          # Recommended for large downloads

Rate Limits

Authentication	Requests/Second	Delay Between
Email only	3	0.34 seconds
Email + API key	10	0.1 seconds

Get an API key at: https://www.ncbi.nlm.nih.gov/account/settings/

History Server

The history server stores search results on NCBI servers, enabling efficient batch retrieval without re-sending large ID lists.

How It Works

1. Search with usehistory='y'
2. Get WebEnv (session ID) and query_key (result set ID)
3. Fetch results in batches using these identifiers
4. Results stay available for ~15 minutes

# Search with history
handle = Entrez.esearch(db='nucleotide', term='human[orgn] AND mRNA[fkey]', usehistory='y')
search = Entrez.read(handle)
handle.close()

webenv = search['WebEnv']
query_key = search['QueryKey']
total = int(search['Count'])

print(f"Found {total} records, stored in history")

Core Pattern: Batch Download

Goal: Download all records matching a search query, handling NCBI rate limits and large result sets.