Token Burn Rate

What it means

Tokens in plus tokens out, multiplied by call frequency and retries, equals burn rate. Long context, verbose prompts, and chained inference multiply it quickly.

Why designers should care

Design choices (auto-expand context, unlimited regenerate, verbose CoT visible by default) directly raise burn. Users need budgets, summaries, and caps that match value.

Example

An agent dashboard shows “~12k tokens this task” with a breakdown by step; users can switch to a compact mode that summarizes tool results before the next LLM call.

Common mistakes

• • Agent UIs that silently re-run full context on every minor retry.
• • No session or org-level visibility until users hit hard rate limits.
• • Optimizing latency without measuring token cost per successful task completion.

Related terms

• Token: Models process and bill text as tokens; a rough rule is ~¾ of a word per token in English, but code and symbols can consume more.
• Context Window: Everything sent to the model must fit inside the context window; overflow is truncated, summarized, or rejected depending on product policy.
• Inference: Each user request triggers inference: tokens in, tokens out, optionally interleaved with tool calls and retrieval.
• Latency: Network, queue, retrieval, tool calls, and generation length all add wait time before users can read, edit, or act on output.
• Agent: The model decides sequences of actions (search, draft, click, call APIs) within policies until the task completes or hits a human checkpoint.