requests

requests Best Practices

requests is the definitive HTTP client for Python. To build robust, performant, and maintainable web interactions, you must leverage its advanced features and integrate them with modern Python best practices. This guide provides actionable rules for our team.

1. Always Use Session Objects

For any code making more than a single HTTP call, or any reusable API client, you must use a requests.Session object. Sessions provide connection pooling and cookie persistence, drastically improving performance and resource usage.

❌ BAD: Direct requests calls

import requests

# Each call creates a new connection
response1 = requests.get("https://api.example.com/data/1")
response2 = requests.get("https://api.example.com/data/2")

✅ GOOD: Use a Session with a context manager

import requests

with requests.Session() as session:
    # Connections are pooled and reused
    response1 = session.get("https://api.example.com/data/1")
    response2 = session.get("https://api.example.com/data/2")

2. Configure Retries and Timeouts with HTTPAdapter

Enhance session reliability by mounting a custom HTTPAdapter to handle retries with backoff and set default timeouts. This prevents flaky network issues from crashing your application and ensures requests don't hang indefinitely.

❌ BAD: No retries, no default timeouts

import requests

with requests.Session() as session:
    # Will fail on first network glitch, can hang forever
    response = session.get("https://api.example.com/flaky-endpoint")

✅ GOOD: Mount an HTTPAdapter with Retry logic

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

# Configure retry strategy
retry_strategy = Retry(
    total=3,  # Total number of retries
    backoff_factor=1,  # Exponential backoff (1, 2, 4 seconds)
    status_forcelist=[429, 500, 502, 503, 504], # HTTP statuses to retry on
    allowed_methods=["HEAD", "GET", "OPTIONS"] # Methods to retry
)
adapter = HTTPAdapter(max_retries=retry_strategy)

with requests.Session() as session:
    session.mount("http://", adapter)
    session.mount("https://", adapter)

    # All requests made with this session will use the retry logic
    response = session.get("https://api.example.com/flaky-endpoint", timeout=(5, 10))

Note: timeout should still be explicitly set on individual requests, even with an adapter, to override or confirm the default.