asyncopenai-concurrency-httpx-pool

AsyncOpenAI Concurrency: the Hidden httpx Pool Cap

Problem

Async batch scorers typically gate concurrency with asyncio.Semaphore(N). Raising N above ~100 silently does nothing: the OpenAI SDK's default httpx transport caps the connection pool at max_connections=100, so excess tasks queue inside httpx instead of reaching the provider. The semaphore looks like the throttle but is not the binding constraint — there is no error, just a throughput ceiling.

Context / Trigger Conditions

• asyncio.Semaphore(N) with N > 100 around client.chat.completions.create shows the same throughput as N = 100
• Client constructed as AsyncOpenAI(api_key=..., base_url=...) with no http_client argument (the default transport)
• Provider is known to allow high concurrency (DeepSeek v4-flash: ~2500)
• Symptom check: requests-in-flight measured at the server never exceeds ~100

Solution

Size the httpx pool to the semaphore when constructing the client:

import httpx
from openai import AsyncOpenAI

CONCURRENCY = 2000
client = AsyncOpenAI(
    api_key=..., base_url="https://api.deepseek.com",
    http_client=httpx.AsyncClient(limits=httpx.Limits(
        max_connections=CONCURRENCY,
        max_keepalive_connections=CONCURRENCY)))
sem = asyncio.Semaphore(CONCURRENCY)

Both edits are required; either alone caps the other. Keep the per-request retry loop — at high concurrency transient failures are more likely, and the retry envelope is what turns them into non-events.

Verification

Throughput scales with N. Verified 2026-07-17 on DeepSeek v4-flash (deepseek-chat, JSON-mode unit scoring, ~1.5k-token prompts): at semaphore 50 a cold 13.8k-request chunk took ~70 min; at semaphore 2000 + matched pool, a 14.4k-request chunk took ~11 min (~40 req/s sustained, ~250/s burst on a 1.7k-request tail chunk), 0 failed requests, 0 schema-invalid responses. Effective speedup ~7x rather than 40x — server-side queuing absorbs the rest — but with zero reliability cost.