ai-cost-token-optimizer

AI Cost & Token Optimizer

English | Bahasa Indonesia

---

English

Purpose & Overview

Production-grade guidelines for FinOps in AI engineering — prompt caching (Anthropic Prompt Caching, Gemini Context Caching), dynamic model routing (routing lightweight queries to Flash/Haiku and complex reasoning to Pro/Opus), semantic caching with Redis/GPTCache, and real-time token expenditure tracking.

Key Capabilities

• Prompt & Context Caching: Storing static system prompts, long-context documents, and schemas in cache to reduce token costs by up to 90%.
• Model Router: Heuristic and classifier-based routing between ultra-fast Flash models and high-reasoning Pro models.
• Semantic Caching: Hashing query vector embeddings to serve cached responses for semantically identical user queries.

// Model Routing Strategy Example
export function selectOptimalModel(promptLength: number, taskType: 'classification' | 'reasoning' | 'summary') {
  if (taskType === 'classification' || promptLength < 500) {
    return 'gemini-3.5-flash'; // High speed, ultra low cost
  }
  return 'gemini-3.1-pro'; // Complex reasoning
}

Implementation Checklist

• Enable Context Caching for static system prompts or documents larger than 32k tokens.
• Implement a router heuristic: use gemini-3.5-flash for simple parsing and gemini-3.1-pro for deep reasoning.
• Set up semantic caching (e.g., Redis + Vector Search) for frequently asked identical queries.
• Monitor token usage and set hard budgeting limits per user/tenant to prevent abuse.

Operating Protocol

1. Model Fallback & Routing: Sets up an abstraction layer (like LiteLLM or Vercel AI SDK Core) to support multiple providers.
2. Complexity Scoring: Implements heuristics (prompt length, required JSON schema, keyword analysis) to route to the cheapest capable model.
3. Semantic Caching Integration: Implements a Vector DB or Redis caching layer. Before routing to an LLM, it embeddings the user prompt and checks if a semantically similar query was answered recently.
4. Token Budgeting: Sets hard limits and alerts for daily API consumption per tenant/user.