llm-rate-limiting

LLM Rate Limiting

Implement robust rate limiting to prevent quota exhaustion and handle API limits gracefully.

When to Use

• Hitting API rate limits
• Managing concurrent requests
• Preventing quota exhaustion
• Implementing fair usage policies
• Handling burst traffic

API Rate Limits

Rate limits depend on your provider, usage tier, model, and sometimes region, and providers change them. Read the current values from the official pages and from response headers instead of hard-coding them:

Provider	Official rate-limit docs	Runtime signal
Anthropic	https://platform.claude.com/docs/en/api/rate-limits	anthropic-ratelimit-* response headers, retry-after on 429
OpenAI	https://platform.openai.com/docs/guides/rate-limits	x-ratelimit-* response headers

The tables below are an illustrative shape only (made-up numbers) showing the dimensions to configure: requests per minute, input/output tokens per minute, per tier.

Tier	Requests/min	Input tokens/min	Output tokens/min
Example tier A	50	30K	8K
Example tier B	1,000	450K	90K

Rate Limiter Implementation

Token Bucket Algorithm

class TokenBucket {
  private tokens: number;
  private lastRefill: number;

  constructor(
    private capacity: number,      // Max tokens
    private refillRate: number,    // Tokens per ms
  ) {
    this.tokens = capacity;
    this.lastRefill = Date.now();
  }

  private refill(): void {
    const now = Date.now();
    const elapsed = now - this.lastRefill;
    const newTokens = elapsed * this.refillRate;

    this.tokens = Math.min(this.capacity, this.tokens + newTokens);
    this.lastRefill = now;
  }

  async acquire(tokens: number = 1): Promise<boolean> {
    this.refill();

    if (this.tokens >= tokens) {
      this.tokens -= tokens;
      return true;
    }

    return false;
  }

  async waitForTokens(tokens: number = 1): Promise<void> {
    while (!(await this.acquire(tokens))) {
      const waitTime = (tokens - this.tokens) / this.refillRate;
      await sleep(Math.min(waitTime, 1000)); // Max 1 second wait
    }
  }
}

// Usage
const limiter = new TokenBucket(
  1000,  // 1000 tokens capacity
  1000 / 60000  // 1000 tokens per minute = ~16.67 per second
);

async function makeRequest() {
  await limiter.waitForTokens(1);
  return callAPI();
}