ai-cost-audit

AI Cost Audit

The Law

EVERY LLM FEATURE HAS A COST TRAJECTORY. DISCOVER IT BEFORE 10× SCALE DISCOVERS YOU.
"It's cheap enough now" is a claim about current volume, not future volume.
"The API has reasonable pricing" is not a projection.
Token counts + call volume + cost at 10× scale IS a cost audit.

When to Use

Trigger:

• Before launching any LLM feature (pre-launch projection)
• When monthly API bill increased > 20% with no obvious cause
• Before scaling a feature to a new user segment
• Before committing to a model or provider for a high-volume use case

When NOT to Use

• Internal one-off scripts or developer tools with < 50 calls/day — cost is negligible; write the call, move on
• Features still in prototype where the call structure will change significantly before launch — audit after the design stabilises
• When total monthly API cost is guaranteed < $50 regardless of 10× scale — skip the audit, check the bill quarterly

The Process

Step 1 — Count Tokens Precisely

Do not estimate. Count:

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")  # cl100k for GPT/Claude

def count_tokens(text: str) -> int:
    return len(enc.encode(text))

# Measure each segment separately
print("System prompt:", count_tokens(system_prompt))
print("Avg context:", count_tokens(avg_context_sample))
print("Avg user message:", count_tokens(avg_user_message_sample))
print("Avg output:", count_tokens(avg_output_sample))

Get real samples from logs or representative test data — not the "hello world" example.

Step 2 — Measure Call Volume

Calls per user session: N
Sessions per day: M
Background/batch calls per day: K
Retry rate: R% (from logs or estimate)
Total calls per day: (N × M) + K × (1 + R/100)

Step 3 — Calculate Current Cost

COST_PER_1K_INPUT  = 0.003   # $/1k tokens — replace with actual model pricing
COST_PER_1K_OUTPUT = 0.015

def cost_per_call(input_tokens, output_tokens):
    return (input_tokens / 1000 * COST_PER_1K_INPUT
          + output_tokens / 1000 * COST_PER_1K_OUTPUT)

daily_cost    = cost_per_call(avg_input, avg_output) * calls_per_day
monthly_cost  = daily_cost * 30