Prefill vs Decode

See why a model reads your prompt fast but writes its answer slowly: the first is limited by the GPU's arithmetic, the second by how fast it can read its own weights. Nothing is uploaded.

Model and GPU

Workload

Assumptions (change if you know better)

Estimates. They count arithmetic on the weights and bytes read, and leave out attention arithmetic, overheads and queueing.

What limits each phase

The two phases side by side

PrefillDecode (one step)

Time to first token as the prompt grows

Tokens a second as the batch grows

How it is worked out

    What it is for

    How it works

    Answering a prompt has two phases. In prefill the model reads the whole prompt in one go and builds the KV cache. In decode it writes the answer one token at a time, each token needing a pass through the whole model. The first token appears when prefill ends, so prefill sets the time to first token. Decode sets the speed of everything after it.

    The two phases stress a GPU in opposite ways. A GPU has two limits: how many arithmetic operations it can do each second, and how many bytes it can read from its memory each second. During prefill a prompt of thousands of tokens shares one read of the weights, so there is a lot of arithmetic for every byte, and the arithmetic limit is reached first. During decode one token needs all the weights read again for very little arithmetic, so the memory limit is reached first. The first is called compute-bound, the second memory-bound.

    The roofline chart shows this. The horizontal axis is arithmetic intensity: operations done for every byte moved. The sloped part of the line is where memory is the limit; the flat part is where compute is. The bend between them is the ridge. Decode sits far to its left. Prefill sits to its right, unless the prompt is so short that it too has little arithmetic to do.

    Batching moves decode to the right. The weights are read once for the whole batch, so a bigger batch means more operations per byte read, and the total tokens per second rise almost for free until the GPU's arithmetic limit. Each sequence also needs its own cache read, which is why long contexts keep decode memory-bound at larger batches.

    This is an estimate, not a benchmark. It counts the arithmetic on the weights (2 operations per parameter per token) and the bytes of weights and cache read, applies an efficiency to each limit, and takes whichever phase component is slower. It leaves out attention arithmetic, kernel launch overheads, communication and scheduling. Real servers reach somewhere between half and the whole of these figures, which is why the two efficiencies are yours to change.

    Questions

    What is prefill?

    The first phase of answering. The model reads the entire prompt at once, computes the keys and values for every token, and stores them in the KV cache. The time it takes is the time to first token.

    What is decode?

    The second phase. The model writes the answer one token at a time. Each new token needs the whole model and the cache read from memory once, so decode is limited by memory bandwidth rather than arithmetic.

    Why is prefill faster per token than decode?

    Because the prompt tokens are processed together. One read of the weights serves thousands of tokens, so the GPU's arithmetic units stay busy. In decode one read of the weights produces one token per sequence.

    What is a roofline chart?

    A chart of the speed a program can reach against its arithmetic intensity, the operations it does for every byte it moves. The sloped part is the memory limit and the flat part the compute limit. It shows at a glance which limit a workload hits.

    What does time to first token depend on?

    The prompt length, the model size and the GPU's compute. It also includes any wait in a queue, which this page does not model. Retrieved chunks lengthen the prompt, so they lengthen the wait.

    Why does batching help decode but not prefill?

    A prefill already shares one weight read across thousands of tokens, so its arithmetic is busy. A decode step with one sequence shares nothing, so adding sequences gets more useful work out of the same read.

    Where do the TFLOPS and bandwidth figures come from?

    The GPU datasheets. The compute figure is dense FP16, without the sparsity doubling that vendors sometimes quote. Real kernels reach part of each figure, so the page applies an efficiency, 50% for compute and 80% for memory by default. Change them under Assumptions.

    Can I share my settings?

    Yes. The settings are kept in the page address, so Copy link gives a link that opens the page exactly as you left it. CSV and JSON download the inputs and the results.

    Is anything uploaded?

    No. The page is a calculator that runs in your browser tab. The numbers you enter stay on your device.