Speculative decoding, explained

By TryRAG · Updated

Try it: the Speculative Decoding Simulator runs draft-and-verify rounds and maps the speedup for every draft length and acceptance rate.

A large language model writes one token per pass through the network, and each pass is slow. Speculative decoding makes it faster without changing what it writes. A small model guesses several tokens ahead, and the large model checks all the guesses in one pass.

The idea

Much of any text is easy to predict. After "The capital of France is" almost any model will say "Paris". Using the large model for such tokens is a waste. But nobody knows in advance which tokens are easy, so the small model guesses all of them and the large model decides which guesses to keep.

One round, step by step

  1. Draft. A small, fast model writes the next few tokens, say four.
  2. Verify. The large model, called the target, reads all four in a single pass. A model can check many positions at once in about the time it takes to write one token.
  3. Accept. The target keeps the draft tokens up to the first one it disagrees with.
  4. Correct. At that point it supplies its own token, and the rest of the draft is discarded. If it agreed with all four, it adds a fifth of its own.

Every round therefore produces between one and five tokens for the cost of one large pass and four small ones. The acceptance rule is designed so that the result follows the target model's own choices: the text is the same as if the target had written it alone.

How much faster

Call the draft length k, the chance that a draft token is accepted α, and the cost of one draft pass compared with one target pass c. If each token is accepted independently:

Tokens per round = (1 − αk+1) ÷ (1 − α)
Speedup ≈ tokens per round ÷ (1 + k × c)

Draft is rightTokens per round (k = 4)Speedup (c = 0.1)
90% of the time4.102.93×
80%3.362.40×
60%2.311.65×
20%1.250.89×, slower

When it does not help

Words you will meet