llm-finops-router

LLM FinOps & Dynamic Model Router

English | Bahasa Indonesia

---

English

Orchestration & Integration

Connects and orchestrates with:

• ai-llm-integration-expert — base LLM integration patterns
• vercel-ai-sdk-expert — streaming + tool-call routing on the edge
• multi-agent-orchestration — per-agent budget allocation
• cloud-hosting-expert — gateway deployment (Vercel, Cloudflare, AWS)
• llm-observability-expert — Langfuse traces + cost dashboards
• adaptive-model-cascade — complexity-scored model selection
• kv-cache-prefix-optimizer — KV-cache hit maximization to cut costs

---

Purpose

Optimize operational costs, reliability, and latency of multi-model LLM architectures using dynamic routing, provider fallback chains, semantic caching, and real-time FinOps analytics. Target: 40–70% reduction in monthly LLM spend without degrading output quality.

---

1. LiteLLM v1.x — Deep Configuration Guide

LiteLLM provides a unified OpenAI-compatible interface across 100+ LLM providers with built-in fallback, retries, and budget management.

Install & Start Proxy

pip install litellm[proxy]
litellm --config litellm_config.yaml --port 4000

litellm_config.yaml — Full Production Config

model_list:
  - model_name: fast-tier            # Virtual name used by clients
    litellm_params:
      model: gemini/gemini-3.8-flash
      api_key: os.environ/GEMINI_API_KEY
      rpm: 1000
      tpm: 2000000

  - model_name: fast-tier            # Fallback 1 for fast-tier
    litellm_params:
      model: groq/llama-3.1-8b-instant
      api_key: os.environ/GROQ_API_KEY
      rpm: 600

  - model_name: reasoning-tier
    litellm_params:
      model: anthropic/claude-5-sonnet
      api_key: os.environ/ANTHROPIC_API_KEY
      rpm: 100
      tpm: 400000

  - model_name: reasoning-tier       # Fallback 1 for reasoning-tier
    litellm_params:
      model: gemini/gemini-4-pro
      api_key: os.environ/GEMINI_API_KEY
      rpm: 60

  - model_name: embedding-tier
    litellm_params:
      model: text-embedding-3-small
      api_key: os.environ/OPENAI_API_KEY

router_settings:
  routing_strategy: least-busy        # Options: simple-shuffle | least-busy | latency-based-routing | cost-based-routing
  enable_pre_call_checks: true
  num_retries: 3
  retry_after: 2                      # seconds between retries
  timeout: 60
  fallbacks:
    - fast-tier: [groq/llama-3.1-8b-instant, openai/gpt-4o-mini]
    - reasoning-tier: [gemini/gemini-4-pro, openai/gpt-5]
  context_window_fallbacks:
    - reasoning-tier: [gemini/gemini-4-pro]  # 2M+ context fallback

litellm_settings:
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]
  cache: true
  cache_params:
    type: redis
    host: os.environ/REDIS_HOST
    port: 6379
    password: os.environ/REDIS_PASSWORD
    ttl: 600                          # Cache TTL in seconds
    similarity_threshold: 0.92        # Semantic cache hit threshold

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL  # PostgreSQL for spend tracking

# Virtual Keys with per-user budget limits
virtual_keys:
  - key: sk-agent-research
    user_id: agent-research
    max_budget: 5.00                  # USD per day
    budget_duration: 1d
    allowed_models: [fast-tier, embedding-tier]
  - key: sk-agent-code
    user_id: agent-code
    max_budget: 10.00
    budget_duration: 1d
    allowed_models: [fast-tier, reasoning-tier]