How LLM inference works

By TryRAG · Updated

Try it: the inference playgrounds are seven pages to work through in order. Start with Prefill vs Decode.

Retrieval, the subject of most of this site, decides what goes into the prompt. Inference is what happens after that: a model running on a GPU reads the prompt and writes the answer. This page is the map. It follows one request and shows where each idea fits.

One request

One requestPromptquestion + chunksRead the promptonce: fills the cacheWrite a tokenrepeat until doneAnswerone pass per token4GPU memoryWeights (shared by everyone)cache 1cache 2cache 3cache 4free213One KV cache for every conversation being served. More caches that fit means a bigger batch.1KV cache: memory for one conversation2Quantization: smaller weights3Batching: many conversations in each pass4Speculative decoding: several tokens a pass5Cost: tokens per second and how busy the GPU is, turned into money
Top: what happens to one request. Bottom: what is in GPU memory while it happens. The numbers match the five steps below.

A request has two phases. First the model reads the prompt, all at once. Then it writes the answer one token at a time: each new token needs one pass through the whole model, and each pass looks back at everything before it. Writing is the slow part, because a 300-token answer is 300 passes.

Two things sit in GPU memory while this happens. The weights are the model itself, loaded once and shared by every request. The KV cache is the model's stored working for one conversation, and every conversation being served needs its own.

The first playground, Prefill vs Decode, shows why these two phases are limited by different things: the first by the GPU's arithmetic, the second by how fast it can read the model from memory. The five ideas below build on that.

The five ideas, in order

  1. The KV cache decides what fits. It grows with the length of the prompt and answer, and it is multiplied by the number of conversations served at once. On a busy server it can be larger than the model. KV Cache Calculator · guide
  2. Quantization shrinks the weights. Storing each weight in 4 or 8 bits instead of 16 makes the model a quarter or half the size, at the price of a small rounding error. What it frees can hold more cache. Quantization Visualizer · guide
  3. Batching uses the room. A GPU can write the next token for many conversations in one pass, almost as fast as for one. How the server fills and refills that batch decides how many tokens per second it really gets. Batching Simulator · guide
  4. Speculative decoding speeds up one answer. A small model guesses several tokens ahead and the large model checks them in a single pass. This is a separate idea from batching: it makes each answer arrive sooner. Speculative Decoding Simulator · guide
  5. Cost follows from the rest. A GPU is paid for by the hour. Tokens per second, and the share of the time the GPU is actually busy, turn that into a cost per token that can be compared with an API price. Inference Cost Calculator · guide

Two more playgrounds sit alongside these: Prefill vs Decode (guide), which explains the two phases of a request, and the Prefix Caching Simulator (guide), which shows how much of a prompt a server can avoid reading twice.

How they connect

The five are one chain. Memory is fixed, so smaller weights (2) leave more room for cache (1). More room for cache means more conversations in a batch (3). A bigger batch, and more tokens per pass (4), mean more tokens per second. More tokens per second from the same hourly bill means a lower cost per token (5), as long as there is enough traffic to keep the GPU busy.

Each playground ends with a link to the next and carries your numbers across where they apply: the model and GPU you chose in the KV calculator open in the quantization page, and the speed-up you find in the speculative decoding page is applied in the cost calculator.

Where RAG comes in

Retrieved chunks are part of the prompt. A longer prompt takes longer to read, fills more cache for every conversation, leaves room for fewer conversations in a batch, and costs more on an API that charges for each prompt token. That is why chunk size and top-k matter for cost as well as for answer quality. Tick "Add retrieved chunks" on the KV cache, batching and cost pages to see the effect.

Words you will meet