Continuous batching, explained
By TryRAG · Updated
Try it: the Batching Simulator runs the same requests through static batching, continuous batching and continuous batching with a paged KV cache, side by side.
A GPU can work out the next token for many requests in one step, in nearly the time it takes for one. Running requests together, in a batch, is therefore the main way a model server gets more out of its hardware. How the batch is filled decides how much of that gain is real.
Static batching
The simple approach: wait for a few requests, run them as a batch, return all the answers when the batch is finished. The trouble is that answers have different lengths. If one answer needs 200 tokens and seven need 20, the seven are finished after 20 steps and their places in the batch do nothing for the other 180. The whole batch, and everyone waiting behind it, is held up by its longest member.
Continuous batching
Continuous batching looks at the batch after every single step.
- StepEvery request in the batch gets its next token.
- CheckRequests that have finished leave straight away.
- RefillWaiting requests take the free slots.
- RepeatThe next step runs with a full batch again.
No slot is idle while requests are waiting, and a short request is not delayed by a long one that happened to arrive with it. On a mixed workload this often gives several times the throughput of static batching and much lower waiting times. It is what servers such as vLLM and Text Generation Inference do.
The memory limit
Each running request needs KV cache memory, and the size of the batch is limited by how many caches fit. A server that does not know how long an answer will be can play safe and reserve room for the longest answer it allows. Most answers are far shorter, so most of that reserved memory holds nothing, and the batch is smaller than it could be.
Paged KV cache
A paged cache treats GPU memory the way an operating system treats RAM. Memory is divided into small blocks. A request gets a block when it needs one and another when that is full. Nothing is reserved for tokens that have not been written. The waste falls from "the gap between each answer and the maximum" to "the unused part of each request's last block", and more requests fit in the same memory. This is the idea known as PagedAttention.
| Static | Continuous | Continuous + paged KV | |
|---|---|---|---|
| Finished request | Waits for the batch | Leaves at once | Leaves at once |
| Free slot | Idle until the next batch | Refilled next step | Refilled next step |
| Cache memory | Reserved for the maximum | Reserved for the maximum | Given in blocks as needed |
Long prompts and chunked prefill
So far a prompt has been treated as read in one step. A long prompt is not. Reading 16,000 tokens takes seconds, and while the server reads them, nobody else gets a token. For every other user the answer freezes. Chunked prefill avoids this by reading the long prompt a piece at a time, with a decode step for everyone else between pieces. With 512-token chunks the other users wait about a tenth of a second at a time instead of three seconds.
The cost is that every pass also does a decode step, so the long prompt itself reaches its first token later: in the Batching Simulator's example, 4.2 seconds instead of 3.2. Smaller chunks mean shorter pauses for the others and a longer wait for the long prompt. Real servers such as vLLM and SGLang let you set the chunk size. See prefill and decode for why reading a prompt and writing an answer behave differently.
What it means for RAG
RAG prompts are long and the answers are usually short. The retrieved chunks sit in the cache for the whole request, so long prompts reduce how many requests fit in a batch. Fewer or shorter chunks mean a larger batch and more answers per second from the same GPU.
Words you will meet
- Throughput: tokens written per second across all requests.
- Latency: the time one request takes from arrival to its last token.
- p95 latency: the time within which 95 of 100 requests finish.
- Preemption: pausing a request and sending it back to the queue when memory runs out.