token-cost-tracking

Token and Cost Tracking

Track token usage and costs across agents for budget management and optimization.

Core Principle

Every organization needs to answer:

1. How much are we spending on LLMs?
2. Where is the spend going (features, agents, users)?
3. Are we within budget?
4. What's trending (up or down)?

Essential Attributes

# Per-call token tracking
span.set_attribute("llm.tokens.input", 1500)
span.set_attribute("llm.tokens.output", 350)
span.set_attribute("llm.tokens.total", 1850)

# Per-call cost
span.set_attribute("llm.cost_usd", 0.025)

# Attribution
span.set_attribute("cost.feature", "document_analysis")
span.set_attribute("cost.agent", "researcher")
span.set_attribute("cost.user_id", "user_abc")  # Hashed
span.set_attribute("cost.org_id", "org_123")

Model Pricing Table

Keep pricing updated (prices as of late 2024):

MODEL_PRICING = {
    # Anthropic (per 1M tokens)
    "claude-3-opus": {"input": 15.00, "output": 75.00},
    "claude-3-5-sonnet": {"input": 3.00, "output": 15.00},
    "claude-3-5-haiku": {"input": 0.80, "output": 4.00},
    "claude-3-haiku": {"input": 0.25, "output": 1.25},

    # OpenAI (per 1M tokens)
    "gpt-4-turbo": {"input": 10.00, "output": 30.00},
    "gpt-4o": {"input": 2.50, "output": 10.00},
    "gpt-4o-mini": {"input": 0.15, "output": 0.60},
    "gpt-3.5-turbo": {"input": 0.50, "output": 1.50},

    # Embeddings (per 1M tokens)
    "text-embedding-3-large": {"input": 0.13, "output": 0},
    "text-embedding-3-small": {"input": 0.02, "output": 0},
    "text-embedding-ada-002": {"input": 0.10, "output": 0},
}

Cost Calculation

def calculate_cost(
    model: str,
    input_tokens: int,
    output_tokens: int,
    cached_tokens: int = 0,
    cache_discount: float = 0.9  # 90% discount for cached
) -> float:
    """Calculate cost for an LLM call."""
    pricing = MODEL_PRICING.get(model)
    if not pricing:
        return 0.0

    # Cached tokens are discounted
    effective_input = input_tokens - cached_tokens
    cached_cost = (cached_tokens / 1_000_000) * pricing["input"] * (1 - cache_discount)
    input_cost = (effective_input / 1_000_000) * pricing["input"]
    output_cost = (output_tokens / 1_000_000) * pricing["output"]

    return round(input_cost + cached_cost + output_cost, 6)