home-ops: Skill for Claude Code

LiteLLM proxy, operator CRs and the auto-router

Relocated verbatim from AGENTS.md on 2026-09-01 so it loads only when this subsystem is in play. The text below is unchanged; only line breaks were inserted. AGENTS.md keeps a one-sentence pointer. Add new findings here or to the owning document, not back into AGENTS.md - see its "Maintaining this file" section for the rule.

• LiteLLM has no config-file-only way to declare a virtual key with a budget or rate limit - keys with max_budget/rpm_limit/tpm_limit only exist via the proxy's /key/generate REST API, and without a connected Postgres DATABASE_URL that API stores keys in memory only (gone on every pod restart). This is why ai/litellm (governance layer, docs/ai-system/litellm/README.md) depends on the shared postgres-17 CNPG cluster even though the rest of ai/ is stateless-by-preference, and why its virtual keys are minted through that API rather than declared in config.yaml. Since captain decision O1 (2026-08-26) the minting is done by the home-operations litellm-operator (kubernetes/apps/base/ai/litellm-operator/, chart pinned in its app/helmrelease.yaml): LiteLLMVirtualKey CRs in ai/litellm/app/virtualkeys/ replaced a hand-written consumers.json + provision_keys.py + hook Job + CronJob + RBAC, and LiteLLMModel CRs in app/models/ replaced the config.yaml ConfigMap.

Eleven things to know before touching it:

(1) LiteLLMProxySpec has no securityContext, serviceAccountName, automountServiceAccountToken, strategy, initContainers or startupProbe field, so the operator-rendered proxy pod carries none of this repo's usual pod hardening and the default SA token is mounted - the accepted-gap table and why a kustomize patch cannot fix it (the operator reconciles that Deployment back) are in kubernetes/apps/base/ai/litellm/README.md.

(2) Because there are no initContainers, DB bootstrap is a standalone postgres-init Job (app/dbinit.yaml), not a CNPG Database CR: that CRD's spec.owner is required and CNPG does not create the role, which would force spec.managed.roles edits on the shared postgres-17 Cluster.

(3) The operator's ValidatingWebhookConfiguration is failurePolicy: Fail for every LiteLLM kind, so ai/litellm must dependsOn ai/litellm-operator with a Deployment healthCheck - with the operator down the API server rejects the CRs rather than leaving them unreconciled.

(4) LiteLLM key aliases are globally unique and the operator has no adopt-by-alias path, so a key that already exists under an alias blocks forever the CR that wants it: POST /key/generate returns 400 Key with alias '<name>' already exists. LiteLLMVirtualKeyReconciler only reaches its GetVirtualKey/UpdateVirtualKey branch when its own output Secret exists; with that Secret absent it can only ever retry generate. This bit the O1 cutover live on 2026-08-27 - the retired provision_keys.py keys survived in the Postgres DB the migration deliberately keeps, so both LiteLLMVirtualKeys sat Ready=False/GenerateFailed while the proxy looked perfectly healthy (Ready, 5 models rendered, still serving on the old keys) and the PushSecrets never wrote 1Password. Deleting the colliding key through the admin API (POST /key/delete) is therefore a required step, not tidy-up, whenever a key comes under operator management or an alias is reused. Expect the credential to rotate (the old value starts returning 401) and its accrued spend history to go with it, so treat it as a real rotation and check who holds that key first.

(5) A config-declared fallback in routerSettings BYPASSES the calling virtual key's model allow-list. Measured on v1.98.0: a key scoped to one local-shaped model was served by anthropic/claude-sonnet-5 and billed real USD, having never named it - litellm/router.py and router_utils/ contain no reference to the key's entitlements at all. A caller-SUPPLIED fallbacks array in the request body is checked (403), so only the router's own chain is unchecked. Consequence, and the reason qwen3.6-35b-a3b deliberately has NO fallback while the otherwise-identical chat-ha alias does: a cloud fallback may only be declared on an alias whose every consumer is already cloud-entitled, or captain decision D4's budget/entitlement split is silently void. Full evidence: docs/ai-system/litellm/fallbacks.md.