agent-token-budget

Agent Token Budget

Every agent loop step is another model call, so cost and latency compound. Give each conversation a hard token ceiling and attribute usage to the session — bounding reasoning work, not just dollars.

When to use

• An agentic loop or delegation chain can call the model an unbounded number of times.
• You need to answer "how many tokens did this session use?" and "what happens at the limit?".
• You want token/cost signals in traces and metrics without hardcoding a vendor's prices.

Steps

1. Accumulate usage into session state. After each final model response, add input and output tokens to a running per-session total that persists across turns. Exclude streaming partials.
2. Enforce a hard ceiling before the model call. In a before-model hook, if the session total has reached the limit, short-circuit with a clear message ("start a new session, or raise the limit") instead of a silent failure or an open-ended bill.
3. Attribute, don't just count. Emit tokens as an OTel counter (graph throughput) and as span attributes (per-turn detail); compute cost from configurable per-1k prices that default to 0 for local models — never hardcode a provider's pricing.
4. Make the budget cover the whole conversation. In multi-agent/delegation flows, keep the totals in shared session state so hops between sub-agents accumulate against one ceiling rather than each starting fresh.

Reference implementation

From the AgentOps Open Course, installable with npx skills add MLOps-Courses/agentops-open-course:

• agents/go/policy/budget.go — session usage, hard ceiling, configurable cost estimate, and OTel evidence.
• Course chapters 7.3. Costs and 3.7. Multi-Agent.

Verify

Set the ceiling to 1 token and send a turn: assert the model call is refused with the actionable message and the token counter reflects the usage; confirm a delegation chain hits one shared ceiling, not one per sub-agent.