llm-router-ops

Operating an OpenAI-compatible LLM router

A self-hosted LLM router (e.g. LiteLLM) gives every consumer — human tool or agent — one shared base URL and one shared bearer key, and load-balances or routes to whichever backend actually serves a given model alias. This is the generic client-wiring and operational-gotcha pattern; your own base URL, model aliases, and key-storage location stay in your own inventory.

Minimal client wiring

Every client type needs the same three things: base URL, bearer key, model alias. The alias is what decouples a client from knowing which backend is really serving it.

curl -s https://<router-host>/v1/chat/completions \
  -H "Authorization: Bearer $ROUTER_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "<alias>", "messages": [{"role": "user", "content": "hello"}]}'

from openai import OpenAI
client = OpenAI(base_url="https://<router-host>/v1", api_key=os.environ["ROUTER_MASTER_KEY"])

Any tool with an "OpenAI-Compatible" or "OpenAI API" provider type (chat UIs, low-code agent builders, Ansible-style role group vars) takes the same two values — base URL and key — with no other client-specific config needed.

Adding a backend model

1. Add the model block to the router config: alias name, backend model id, backend base URL.
2. Set the context window explicitly if the router doesn't already know the backend's real native context. An unrecognized backend id otherwise resolves to a null/unset advertised context, which starves any consumer that trusts the router's advertised limit to decide how much to send.
3. Restart/reload the router process so the new config takes effect.
4. Verify: GET /v1/models (with the bearer key) lists the new alias, and its advertised context window is the real one, not null.

Gotcha: env var vs. the tool's own persisted config

Many chat-UI-style front ends persist their provider config in their own database and will silently let that override the environment on a redeploy — so a correct env var stops mattering after the UI's admin panel has ever been touched. If the front end has a "persistent config" or equivalent toggle, disable it so the environment is authoritative on every restart, instead of the tool re-applying whatever was last saved in its UI.