Batching Simulator
Watch static batching, continuous batching and continuous batching with paged KV memory serve the same stream of requests, side by side. Nothing is uploaded.
- 1Press Play
- 2Compare the three servers
- 3Change the arrival rate or the memory budget
Requests and server
A scheduling model, not real GPU performance. Every step takes the same 30 ms whatever the batch size. Compare the three servers with each other.
Static batching
A batch starts together and ends together. Finished requests wait for the longest one.
Continuous batching
A slot that finishes is given to the next waiting request at the very next step.
Continuous batching + paged KV
The same, but cache memory is handed out in small blocks as each answer grows.
Since the start
| Measure | Static | Continuous | Continuous + paged KV |
|---|
A long prompt arrives: chunked prefill
Everything above treats a prompt as read in one step. A long prompt is not: reading 16,000 tokens takes seconds, and while the server reads it, nobody else gets a token. Chunked prefill reads it a piece at a time, with a decode step for everyone else between pieces.
A decode step takes 30 ms, as everywhere on this page. For prompt reading speeds of real models, see Prefill vs Decode.
| Chunk size | Passes | Longest pause for the others | Long prompt's first token | Compared with not chunking |
|---|
What it is for
- See why a server that waits for the longest request in a batch wastes most of its capacity.
- Understand what continuous batching changes and why it lowers waiting time.
- See how reserving cache memory for the longest possible answer limits the batch, and how paging avoids it.
- Try different request mixes: steady, long-tailed, or mostly short with a few very long.
How it works
A GPU can write the next token for many requests in one step, almost as fast as for one. Serving requests in a batch is therefore the main way to get more tokens per second from the same hardware. The question is how the batch is filled.
Static batching collects requests until the batch is full, or until the first one has waited ten steps, then runs them together and returns them together. Answers have different lengths, so the short ones finish early and their slots do nothing until the longest answer in the batch is complete.
Continuous batching, also called dynamic or iteration-level batching, looks at the batch after every step. A request that has finished leaves at once and a waiting request takes its slot. The batch stays full for as long as there is work.
Every running request also needs KV cache memory. A simple server reserves enough for the longest answer it allows, even though most answers are much shorter, so memory runs out before the batch is full. A paged cache, the idea behind vLLM's PagedAttention, gives each request small blocks only as it grows. More requests fit in the same memory.
This is a model of scheduling, not of a real GPU. Every step takes the same time whatever the batch size, the prompt is read in one step, and nothing else on the machine is modelled. The differences between the three servers are real effects; the absolute numbers are not predictions.
Questions
What is chunked prefill?
Reading a long prompt a piece at a time, with a decode step for every other user between pieces. Without it, a long prompt holds the whole server while it is read, and everyone else waits.
Why not always use small chunks?
Each pass pays for a decode step as well as the chunk, so many small chunks make the long prompt itself slower to reach its first token. The chunk size trades a short pause for the others against a longer wait for the long prompt.
What is continuous batching?
A way of serving many requests on one GPU in which a finished request is replaced by a waiting one after every token step, so that slots in the batch are never left idle while requests are waiting.
Why is static batching slower?
A static batch lasts as long as its longest request. All the slots that finished earlier are held and produce nothing. With answers of very different lengths, most of the batch is idle most of the time.
What is paged KV memory?
The KV cache of a request is kept in small fixed-size blocks that are allocated when needed, like pages of memory in an operating system. Nothing is reserved in advance for tokens that may never be written.
What does "wasted KV memory" measure?
Memory that is held for requests but holds no tokens, as an average share of the budget. With up-front reservation it is the gap between each answer's real length and the longest allowed. With paging it is only the unused part of each request's last block.
What happens when the paged server runs out of blocks?
Requests that need a new block wait. If none of them can continue, the newest request is sent back to the queue and starts again later. Real servers do something similar, called preemption.
Are the tokens per second realistic?
No. Every step is given the same fixed time, which a real GPU does not have. Use the numbers to compare the three servers with each other, not to predict a real system.
What do p50 and p95 latency mean?
Half of the requests finish within the median (p50) time, counted from arrival to the last token. 95 in 100 finish within the p95 time, so it shows how bad the slow cases are.
Can I share my settings?
Yes. The settings are kept in the page address, so Copy link gives a link that opens the page exactly as you left it. CSV and JSON download the inputs and the results.
Is anything uploaded?
No. The page is a calculator that runs in your browser tab. The numbers you enter stay on your device.