llm-integration

LLM Integration

Purpose

Integrate a language model into a production application, where the API is slow, rate-limited, occasionally down, and billed per token — none of which the quickstart mentions.

When to Use

• Adding an LLM to a production application.
• An LLM feature that is slow, expensive, or unreliable.
• Handling rate limits, streaming, or provider failover.
• Deciding where the model call belongs in the architecture.

Capabilities

• Streaming responses and partial rendering.
• Retry, backoff, and rate-limit handling.
• Timeouts and cancellation.
• Prompt caching and response caching.
• Multi-provider fallback.
• Token accounting and cost control.

Inputs

• The feature, its latency budget, and its cost budget.
• The provider's rate limits and their actual behavior under load.
• Whether the output is user-facing (stream it) or machine-consumed (do not).

Outputs

• A client with retries, timeouts, and a circuit breaker.
• Streaming where a user is waiting.
• Cost and latency instrumented per call.

Workflow

1. Stream anything a human waits for — A 12-second response that starts rendering at 400ms feels fast. The same response delivered at once feels broken. Streaming is a perceived-latency fix, not a throughput one.
2. Handle rate limits properly — Honor Retry-After. Exponential backoff with jitter. A retry storm against a rate-limited endpoint extends the outage.
3. Set a timeout — LLM calls can hang. An unbounded call holds a connection and a worker until something else breaks.
4. Cache the stable prefix — Prompt caching makes a large system prompt nearly free after the first call. This is often the largest single cost reduction available.
5. Fail over deliberately — A second provider or a smaller model as a fallback. Decide in advance whether a degraded answer is better than no answer for this feature.
6. Instrument tokens and cost per call — Attributed to the feature and the tenant. Without this, an LLM bill is an unexplainable number.