The Wait Before the First Word Appears
Two bottlenecks live inside one request. The pause you feel is prefill. The speed you feel afterwards is decode. Fixing one does nothing for the other — and one popular fix makes the pause worse on purpose.
Core Concept
Time to first token (TTFT) is the latency from sending a request to receiving the first streamed token. It is dominated by prefill: the single pass in which the model processes every prompt token in parallel and fills the key-value cache (KV cache). Everything after that first token is decode — one token at a time — and it is measured by a different number entirely, inter-token latency (ITL).
So “the model is slow” is not one claim, it is two. Prefill is compute-bound and scales with prompt length, so a longer prompt lengthens the silence before anything appears. Decode is memory-bandwidth-bound, so it sets how fast words then stream in. Same request, opposite limiting resources.
Key Components
Prefill
One parallel pass over all prompt tokens; builds the KV cache. Compute-bound. Its cost grows with prompt length — this is the wait.
Decode
Generates one token per step, reusing the KV cache. Memory-bandwidth-bound. Reported as inter-token latency, not as TTFT.
Prefix caching
A repeated system prompt is prefilled once and reused across requests, so its tokens stop being paid for on every call.
Continuous batching
Packs many requests through the GPU together. Raises throughput; your request now waits in a batch, so TTFT gets worse.
How It Works
- Your request arrives and joins the server’s queue. Nothing has been computed yet, and the clock on TTFT is already running.
- Prefill: the whole prompt goes through the model in one parallel pass. Attention keys and values for every prompt token are written into the KV cache. This is arithmetic-heavy — the GPU’s compute is the limit.
- The first token pops out. TTFT stops here. This is the entire span the user experiences as “is it broken?”
- Decode begins: each new token attends to the cached keys and values, so each step reads a large cache but does little maths — memory bandwidth becomes the limit.
- The gaps between those tokens are inter-token latency. Averaging them into “tokens per second” hides step 2 completely, which is why a fast-streaming system can still feel unresponsive.
If TTFT is the problem
- Shorten the prompt — fewer tokens to prefill
- Reuse a prefix cache for the fixed system prompt
- Cut queue wait ahead of prefill
- Adding GPU memory bandwidth will not help
If ITL is the problem
- Quantise weights — fewer bytes read per step
- Batch to amortise weight reads
- Shrink the KV cache being re-read
- Trimming the prompt barely moves this
Two Metrics, Never One
| Prefill / TTFT | Decode / ITL | |
|---|---|---|
| What it measures | Request → first token | Gap between later tokens |
| Bound by | Compute | Memory bandwidth |
| Grows with | Prompt length | Model size, cache size |
| Fixed by | Shorter prompt, prefix cache | Quantisation, batching |
| Batching effect | Worse | Better per-GPU throughput |
The Falsification — How To Be Wrong Here
The classic error is optimising the wrong half. Enable continuous batching and your dashboard improves: tokens per second up, GPUs better utilised, cost per token down. Meanwhile every individual user waits longer for the first word, because their request now queues behind others in the batch. A serving change can raise throughput and degrade the felt experience simultaneously.
The test: report the 99th percentile of both TTFT and inter-token latency — never the mean of one. A mean hides the queueing tail that batching creates, and a single metric cannot show a trade-off. If a change claims a win and you only have one number, you have not measured it.
Where This Shows Up
- Chat assistants. Long system prompts and retrieved context are prefill cost paid on every turn. Prefix caching the fixed part is often the single largest felt speed-up.
- Retrieval-augmented generation. Stuffing more retrieved passages into the prompt buys accuracy with TTFT. That is a real trade, so measure it rather than assume it is free.
- Autocomplete and coding tools. Perceived quality is almost entirely TTFT; a slower total answer that starts instantly usually wins.
- Capacity planning. Serving teams tune batch size against a TTFT service-level objective — throughput alone will always argue for a bigger batch.
- Voice assistants. The first token gates speech synthesis, so prefill delay becomes audible dead air.
Checkpoint — answer before you move on
- A user says the model “takes forever to start, then types fast.” Which phase is the suspect, what resource is it bound by, and which two fixes actually apply? Hint: name the cache that gets filled during that phase.
- Your serving change raised tokens per second by 40% and users complain it feels slower. Explain the mechanism, and name the metric pair that would have caught it before rollout. Hint: something now sits between arrival and prefill.
- Why does quantising weights help inter-token latency much more than it helps time to first token? Hint: one phase is short on arithmetic, the other on bytes moved per step.