LLM & Inference

Why the Bill Counts Two Kinds of Token

Your API invoice has two line items, not one. Prompt tokens are what the model read. Completion tokens are what it wrote. They are priced differently — and the reason is physics, not policy.

Notebook page comparing prompt tokens read in one parallel prefill pass against completion tokens each requiring their own forward pass

Core Concept

A completion token is a token the model generated. A prompt token is one it read. Providers bill them at different rates — typically 2× to 4× more for completion — and the asymmetry is not arbitrary.

Prompt tokens are consumed in one parallel prefill pass: the whole input goes through the network together, matrix-multiplied in bulk, GPU saturated. Completion tokens cannot do that. Every single output token requires its own full forward pass through the entire network, each one attending over everything that came before it. You cannot generate token 40 before token 39 exists.

So the price gap is you paying for sequential work that cannot be parallelised. Reading is a batch job. Writing is a queue.

Key Components

Read

Prompt tokens

System message, chat history, retrieved documents, tool schemas, the user's question. Everything you send. Processed in one prefill pass — cheap per token, but there can be an enormous number of them.

Written

Completion tokens

Everything the model emits. One forward pass each, strictly in order. Expensive per token, and the count is bounded by what you actually asked it to say.

Unit

What a token is

A subword piece, not a word or a character. Rule of thumb: ~4 characters, or about ¾ of a word. "unbelievable" may split into three; a rare name into five.

Hidden

Thinking tokens

Reasoning models emit internal deliberation you are billed for but never see. Your visible answer is 200 tokens; your completion charge may reflect thousands.

How It Works

  1. You send a requestEvery character of your prompt is tokenised into subword pieces. That count is your prompt-token figure — fixed the moment you hit send.
  2. Prefill: one parallel passAll prompt tokens enter the network at once. The attention maths for the whole input is done in bulk, and the key/value cache is built. One pass, however long the prompt.
  3. Decode: one pass per tokenThe model produces token 1. To produce token 2 it runs the whole network again, now attending over the prompt plus token 1. Then again for token 3. Sequential, unavoidable, per token.
  4. The bill splitsPrompt tokens × input rate, plus completion tokens × output rate. Because decode is serial and prefill is parallel, the output rate carries the multiple.
  5. You audit itLog prompt_tokens and completion_tokens per request, multiply each by the actual published rate, and see which term is really dominating.

The Trap — Do the Arithmetic

Say output is billed at 4× input. Compare two requests:

RequestPromptCompletionCost units
Lean prompt, long answer 500 1,000 500 + 4,000 = 4,500
Stuffed RAG context, terse answer 8,000 100 8,000 + 400 = 8,400

The second request generates ten times less text and costs nearly twice as much. The expensive-per-token half was not the expensive half.

Falsification

The cheap-looking half is where the money usually goes A retrieval-augmented app that stuffs eight thousand tokens of context into every call to save a hundred generated tokens is optimising the wrong term. And reasoning models invert the intuition again, billing hidden thinking tokens you never see. Test it, don't assume it: log both counts per request and multiply by the actual published rates — rather than assuming output dominates because output is priced higher.

Real-World Applications

RAG pipelines. Retrieve fewer, better chunks. Cutting 8k of context to 2k saves more than shortening every answer ever will.
Prompt caching. Providers discount repeated prompt prefixes heavily. Put the stable system block first and the variable part last, so the cache can actually hit.
Chat history windows. Unbounded history means the prompt-token term grows with every turn while answers stay the same length. Summarise or truncate.
Reasoning-model budgets. Cap the thinking budget where the API exposes it, and measure real completion counts — the visible answer length tells you nothing.
Model-choice decisions. Compare providers on your measured input:output ratio, not on the headline output price.

Checkpoint — answer before you move on

  1. Why do providers charge more per completion token than per prompt token? Name the mechanism, not the policy. Hint: what happens once for the whole prompt, and what happens once per output token?
  2. An app sends 8,000 prompt tokens and receives 100 completion tokens, at a 4× output multiple. Which term dominates the bill, and by how much? Hint: work it in cost units — you do not need real dollar rates.
  3. How would you falsify the claim "our costs are driven by how much the model writes"? Hint: what two numbers does every API response already hand you, and what must you multiply them by?