llm-rate-limiting

LLM Rate Limiting

Implement robust rate limiting to prevent quota exhaustion and handle API limits gracefully.

When to Use

• Hitting API rate limits
• Managing concurrent requests
• Preventing quota exhaustion
• Implementing fair usage policies
• Handling burst traffic

API Rate Limits (2026)

Anthropic Claude

Tier	Requests/min	Tokens/min	Tokens/day
Free	5	20K	300K
Tier 1	50	40K	1M
Tier 2	1000	80K	2.5M
Tier 3	2000	160K	5M
Tier 4	4000	400K	10M

OpenAI

Tier	RPM	TPM
Free	3	40K
Tier 1	500	200K
Tier 2	5000	450K
Tier 3	5000	800K
Tier 4	10000	2M

Rate Limiter Implementation

Token Bucket Algorithm

class TokenBucket {
  private tokens: number;
  private lastRefill: number;

  constructor(
    private capacity: number,      // Max tokens
    private refillRate: number,    // Tokens per ms
  ) {
    this.tokens = capacity;
    this.lastRefill = Date.now();
  }

  private refill(): void {
    const now = Date.now();
    const elapsed = now - this.lastRefill;
    const newTokens = elapsed * this.refillRate;

    this.tokens = Math.min(this.capacity, this.tokens + newTokens);
    this.lastRefill = now;
  }

  async acquire(tokens: number = 1): Promise<boolean> {
    this.refill();

    if (this.tokens >= tokens) {
      this.tokens -= tokens;
      return true;
    }

    return false;
  }

  async waitForTokens(tokens: number = 1): Promise<void> {
    while (!(await this.acquire(tokens))) {
      const waitTime = (tokens - this.tokens) / this.refillRate;
      await sleep(Math.min(waitTime, 1000)); // Max 1 second wait
    }
  }
}

// Usage
const limiter = new TokenBucket(
  1000,  // 1000 tokens capacity
  1000 / 60000  // 1000 tokens per minute = ~16.67 per second
);

async function makeRequest() {
  await limiter.waitForTokens(1);
  return callAPI();
}