Prefill and decode, explained
By TryRAG · Updated
Try it: Prefill vs Decode works out the time to first token and the tokens per second for a model on a GPU, and draws what limits each phase.
A language model answers in two phases that behave very differently. The first reads your whole prompt. The second writes the answer one token at a time. Most of what people notice about speed, the wait before anything appears and then the steady stream of words, comes from these two phases.
The two phases
Prefill is the reading. The model takes the whole prompt, the question, the instructions and any retrieved chunks, and works through all of it at once. It builds the KV cache for every token and produces the first token of the answer. The time this takes is the time to first token.
Decode is the writing. Each new token needs one pass through the whole model, and the pass cannot start until the previous token exists. The answer therefore comes out one token per pass, and the time per pass sets the tokens per second.
Two different limits
A GPU can do only so much arithmetic per second, and it can read only so many bytes per second from its memory. Which of the two runs out first depends on how much arithmetic there is for each byte read.
In prefill the weights are read once and used for every token of the prompt. A prompt of 2,000 tokens does 2,000 tokens' worth of arithmetic on one read, so the arithmetic is what takes the time. This is compute-bound.
In decode, one read of the weights produces one token. Every token needs all of the model read from memory again, for very little arithmetic. The memory is what takes the time. This is memory-bound.
Numbers for one example
Take Llama 3 8B in 16-bit weights on an H100, with the page's usual assumptions: half of the peak arithmetic and 80% of the peak memory speed are reached.
| Prefill, 2,048-token prompt | Decode, one token | |
|---|---|---|
| Arithmetic | about 33 trillion operations | about 16 billion operations |
| Data read from memory | about 16 GB (the weights) | about 16 GB (the weights again) |
| Time for the arithmetic | about 66 ms | about 0.03 ms |
| Time for the memory | about 6 ms | about 6 ms |
| Time taken | 66 ms, set by arithmetic | 6 ms, set by memory |
So the first token appears after about 66 ms and each later token takes about 6 ms, which is about 160 tokens a second for one user. Per token, prefill is almost 200 times faster than decode, because the prompt shares one read of the weights.
The formulas
- Work: about 2 operations per parameter per token.
- Prefill time = the larger of (2 × parameters × prompt tokens ÷ usable arithmetic speed) and (bytes of weights ÷ usable memory speed).
- Decode time per token = the larger of (2 × parameters × batch ÷ usable arithmetic speed) and (bytes of weights plus every sequence's cache ÷ usable memory speed).
The playground uses exactly these. They leave out the arithmetic of attention itself, which matters more for very long prompts, and overheads such as launching work on the GPU. Real figures are usually somewhat worse, so treat them as a floor for the time.
The roofline chart
A roofline chart plots the speed a GPU can reach against the arithmetic intensity of the work: operations done for every byte moved. To the left of the ridge, speed rises with intensity, because memory is the limit. To the right it is flat, because arithmetic is. Decode sits far to the left, at about one operation per byte. Prefill sits to the right, at about one per token of prompt.
Short prompts, long prompts and batches
- A very short prompt has so little arithmetic that prefill is memory-bound too. On the example above, below about 185 tokens the first token takes about as long as any later one.
- A long prompt makes prefill grow in step with its length: 8 ms for 256 tokens, 66 ms for 2,048, over half a second for 16,000.
- A bigger batch moves decode to the right. The weights are read once for the whole batch, so adding users costs little extra memory traffic and the total tokens per second rise nearly for free. See continuous batching.
- Lower-precision weights mean fewer bytes to read, so decode speeds up almost in proportion. Prefill barely changes, because arithmetic is its limit. See quantization.
What it means for RAG
Retrieved chunks lengthen the prompt, and a longer prompt is more prefill. Five chunks of 300 words add roughly 2,000 tokens, which on the example above is about 65 ms of extra waiting before the first token. The words that follow arrive at nearly the same speed. If the first token feels slow in your RAG system, look at how much you put in the prompt. If the words themselves come slowly, look at memory speed, precision and batching.
Words you will meet
- Time to first token (TTFT): from sending the prompt to the first token of the answer.
- Compute-bound / memory-bound: limited by arithmetic speed / by memory speed.
- Arithmetic intensity: operations done per byte moved.
- Ridge point: the intensity at which a GPU stops being limited by memory and starts being limited by arithmetic.
- TFLOPS: trillions of arithmetic operations per second.