Prefill and decode
A Socratic walk-through of prefill and decode — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why does a model charge more for the words it writes than for the words it reads?
Look at any commercial model's price list and you find two numbers, not one: a price per million input tokens and a price per million output tokens. The output price is several times the input price — commonly three to five times.
The naive reaction is that this is a pricing decision to discourage long answers. But the ratio is remarkably consistent across vendors who compete hard on price, which hints that something physical is behind it. So ask: is a token the model writes genuinely more expensive than a token it reads, when the same weights and the same arithmetic are involved either way?
Reasoning it through
REASONING #Begin with what happens when your prompt arrives. The model must compute an internal representation for every token in it. Here is the thing to notice: token 500 of your prompt does not need to wait for token 499's representation to be finished, because token 499 is already known — you supplied it. Every position can be processed at once.
That is prefill, and it is a single pass over the whole prompt with all the positions in flight simultaneously. On a GPU, that is nearly the best case there is. The weights get loaded from memory once, and then they are used against thousands of token positions before they are put down. The chip's arithmetic units stay busy. Engineers call this compute-bound: throughput is limited by how fast the hardware can multiply, which is what you are paying for.
Now the generation. The first output token comes out of that same pass. But what about the second? To produce it, the model needs the first output token as input — and that token did not exist a moment ago. Output token two cannot be computed in parallel with output token one, because one of them is an input to the other. This is what autoregressive means, and it is not an implementation choice; it follows from the model defining a probability for each token given all the tokens before it.
So decoding is a loop, and each turn of the loop processes exactly one new token. Ask what that does to the hardware. To compute one token, the model must still read every weight it has — all of them, tens or hundreds of gigabytes for a large model — and use each one against a single position instead of thousands. The arithmetic is trivial; the memory traffic is enormous and unchanged. The chip spends nearly all its time waiting for weights to arrive. This is memory-bandwidth-bound, and it is the crux of the whole matter.
Follow that through to the number. If prefill uses each loaded weight against a thousand positions and decode uses it against one, then per token, decode can be hundreds of times less efficient in hardware utilisation. Why, then, is output only a handful of times more expensive rather than a hundred? Because serving systems recover most of the gap by batching: many users' decode steps are run together, so one load of the weights serves dozens of different requests' next tokens at once. Continuous batching, where a finished request is swapped out and a new one slotted in mid-flight, is the standard technique. The residual factor of three to five is roughly what survives after batching has done what it can — though the exact ratio is a commercial figure, and I would not claim any vendor's price is a direct readout of its cost.
One more piece completes the picture. Each decode step needs the attention keys and values for every earlier token. Recomputing them each step would be quadratic waste, so they are computed once and kept: the KV cache. It keeps the per-step work linear, but consumes memory that grows with every token in the conversation — which is why a serving system can only hold so many long conversations at once.
The analogy
THE ANALOGY #Think of a printing press. Setting the type is slow, but once it is set, running a thousand impressions costs almost nothing extra per sheet — the expensive setup is amortised across the run. Prefill is the setup: the press is loaded once and stamps the entire prompt. Decoding is the situation where you must reset the press for every single sheet, because you cannot know what the next sheet says until the previous one has been printed and read.
A press reset is expensive because of physical labour, whereas a decode step is expensive because of data movement — the weights are hauled from memory to the processor and barely used — so the fix is not a faster reset but printing many customers' sheets from one setup, which is exactly what batching does and which has no natural counterpart in the press.
Clarifying the model
THE MODEL #Both prices are per token, but they are prices for two different physical regimes, not two rates for the same work. That is why the user-visible symptoms differ too. The wait before the first word appears — time to first token — is prefill, and it scales with prompt length. The speed at which words then stream out is decode, and it is roughly constant per token regardless of how long your prompt was, because each step's cost is dominated by reading the weights.
It is worth correcting a common inference: people conclude that a long prompt slows down generation. Mostly it does not slow the rate; it lengthens the initial pause and it enlarges the KV cache, which reduces how many requests can share a batch, which raises cost. The effect on your experience is delay at the start, not a stutter throughout.
This also explains why so much inference research targets exactly this bottleneck. Speculative decoding runs a small cheap model ahead to guess several tokens, then verifies them in one pass of the big model — turning serial steps back into a parallel check. Quantisation shrinks the weights so there is less to haul. Both are attacks on memory traffic rather than on arithmetic, which tells you where the pain actually is.
A picture of it
THE PICTURE #How to readRead left to right as elapsed time. The single wide bar in the Prefill row is the whole prompt, however many tokens it holds, done in one pass — its width grows with token count far more slowly than you would expect. The Decode row is the contrast: four bars that cannot overlap, because each needs the one before it to exist first. The Memory row is the reason for the price: one weight read covers the entire prefill bar, but a fresh read sits under every individual decode bar.
What became clearer
WHAT CLEARED #Reading and writing are not two rates for one activity. Reading a prompt is a single parallel pass in which the weights are loaded once and used against every position, so the hardware runs near its arithmetic limit. Writing is a serial loop in which each token must exist before the next can be computed, so the same full sweep of the weights is hauled from memory to produce one token — the chip mostly waiting rather than calculating. Batching many users' decode steps together is what keeps the output price a small multiple rather than a huge one.
Where to go next
ONWARD #- How speculative decoding converts serial decode steps into one parallel verification.
- Why prompt caching makes a long, reused system prompt nearly free on later calls.
Key terms
TERMS #| Term | What it means |
|---|---|
| Prefill | the initial parallel pass over every prompt token, which also populates the cache. |
| Decode | the serial loop producing one output token per step, each conditioned on the last. |
| Autoregressive | generating each token given all preceding ones, which forces the serial ordering. |
| KV cache | stored attention keys and values for earlier tokens, reused each decode step. |
| Memory-bandwidth-bound | limited by data movement from memory to the processor, not by arithmetic speed. |
| Continuous batching | serving many requests' decode steps from one load of the weights, swapping requests in and out mid-flight. |
Every term the collection defines is gathered in the glossary.