Continuous batching, explained

By TryRAG · Updated

Try it: the Batching Simulator runs the same requests through static batching, continuous batching and continuous batching with a paged KV cache, side by side.

A GPU can work out the next token for many requests in one step, in nearly the time it takes for one. Running requests together, in a batch, is therefore the main way a model server gets more out of its hardware. How the batch is filled decides how much of that gain is real.

Static batching

The simple approach: wait for a few requests, run them as a batch, return all the answers when the batch is finished. The trouble is that answers have different lengths. If one answer needs 200 tokens and seven need 20, the seven are finished after 20 steps and their places in the batch do nothing for the other 180. The whole batch, and everyone waiting behind it, is held up by its longest member.

Continuous batching

Continuous batching looks at the batch after every single step.

  1. StepEvery request in the batch gets its next token.
  2. CheckRequests that have finished leave straight away.
  3. RefillWaiting requests take the free slots.
  4. RepeatThe next step runs with a full batch again.

No slot is idle while requests are waiting, and a short request is not delayed by a long one that happened to arrive with it. On a mixed workload this often gives several times the throughput of static batching and much lower waiting times. It is what servers such as vLLM and Text Generation Inference do.

The memory limit

Each running request needs KV cache memory, and the size of the batch is limited by how many caches fit. A server that does not know how long an answer will be can play safe and reserve room for the longest answer it allows. Most answers are far shorter, so most of that reserved memory holds nothing, and the batch is smaller than it could be.

Paged KV cache

A paged cache treats GPU memory the way an operating system treats RAM. Memory is divided into small blocks. A request gets a block when it needs one and another when that is full. Nothing is reserved for tokens that have not been written. The waste falls from "the gap between each answer and the maximum" to "the unused part of each request's last block", and more requests fit in the same memory. This is the idea known as PagedAttention.

StaticContinuousContinuous + paged KV
Finished requestWaits for the batchLeaves at onceLeaves at once
Free slotIdle until the next batchRefilled next stepRefilled next step
Cache memoryReserved for the maximumReserved for the maximumGiven in blocks as needed

Long prompts and chunked prefill

So far a prompt has been treated as read in one step. A long prompt is not. Reading 16,000 tokens takes seconds, and while the server reads them, nobody else gets a token. For every other user the answer freezes. Chunked prefill avoids this by reading the long prompt a piece at a time, with a decode step for everyone else between pieces. With 512-token chunks the other users wait about a tenth of a second at a time instead of three seconds.

The cost is that every pass also does a decode step, so the long prompt itself reaches its first token later: in the Batching Simulator's example, 4.2 seconds instead of 3.2. Smaller chunks mean shorter pauses for the others and a longer wait for the long prompt. Real servers such as vLLM and SGLang let you set the chunk size. See prefill and decode for why reading a prompt and writing an answer behave differently.

What it means for RAG

RAG prompts are long and the answers are usually short. The retrieved chunks sit in the cache for the whole request, so long prompts reduce how many requests fit in a batch. Fewer or shorter chunks mean a larger batch and more answers per second from the same GPU.

Words you will meet