Temperature

Why does making a model's word choice more random make it more useful for some jobs?

There is a setting on every text generator that does something apparently perverse. Turn it up, and the model becomes less likely to choose the word it considers most probable. Turn it to zero, and it always chooses that word. Framed that way, only one of those can be the right answer — why would you ever ask a system to be less likely to say what it thinks is best?

And yet people turn it up, deliberately, for real work. So either they are wrong, or the word "best" in that sentence is doing something dishonest. Which do you suspect?

Let us look at what the number actually does, because it is small and exact and the intuition follows from it.

At each step the model produces a score for every possible next token — a logit. To turn scores into probabilities, they are exponentiated and normalised, which is the softmax. Temperature enters by dividing every score by T before that exponentiation. Divide by a small number and the gaps between scores are magnified, so the leader's probability swells toward one. Divide by a large number and the gaps shrink toward nothing, so all candidates approach equal likelihood. In the limit, temperature zero is simply "always take the top-scoring token", and very high temperature approaches picking uniformly at random.

Notice something the algebra guarantees: temperature never changes the order. The most probable token is the most probable token at every temperature. All that changes is how much probability mass the leader holds relative to the rest — how decisive the choice is, not what it prefers.

The name is not loose metaphor: the expression is the Boltzmann distribution from statistical physics, where T is literally temperature and governs how likely a system is to occupy a state other than its lowest-energy one.

Now, back to "best". The top-scoring token is the model's estimate of the most probable continuation given the text so far. Is the most probable continuation the same as the best answer? Two reasons it is not.

First, greedily taking the local maximum at each step does not produce the most probable sequence — and it has a documented pathology, described by Holtzman and colleagues in 2020: open-ended greedy text tends toward bland, repetitive output. High-likelihood text and good text are measurably different things.

Second, and more usefully: for many tasks you do not need one good output, you need one output that survives a check. If you can evaluate candidates — run the tests, verify the arithmetic, take a majority vote across several attempts — then variety is the raw material of that process. Sampling several reasoning paths and taking the most common final answer, the technique called self-consistency published in 2022, beats greedy decoding on arithmetic and commonsense benchmarks. Not because randomness is wise, but because a spread of attempts plus a filter finds answers one confident attempt does not.

Do you see the shape now? Randomness is not an alternative to correctness. It is a way of buying coverage — and coverage is only worth anything when you have some way of choosing among what it turns up.

A few refinements make this practical.

Temperature is rarely used alone. It is usually paired with a truncation rule — top-k, keeping only the k highest-scoring candidates, or top-p (nucleus sampling), keeping the smallest set of candidates whose probabilities sum to p. The order matters: truncation discards the long tail of implausible tokens, and temperature then reshapes what remains. This is why high temperature does not produce the gibberish the raw algebra suggests: the absurd candidates were removed before the flattening was applied.

Two misconceptions are worth correcting. The first is that temperature zero means accurate. It means deterministic and most-probable, which is a claim about the model's beliefs and not about the world — a confidently wrong answer is delivered just as reliably at temperature zero, and arguably more so, since you lose any chance of noticing that a resampling would have given something different. The second is that temperature zero means reproducible. In a shared serving environment it usually does not, quite: the order of floating-point reductions on a GPU depends on how requests were batched together, and that can flip a near-tie between two tokens.

So how do you choose? By asking what you would do with a second attempt. Where the task has one correct output and no way to check it — extraction into a schema, a tool call, a classification — variety has no consumer, and low temperature is right. Where the task has many acceptable outputs and a human will pick — naming, brainstorming, drafting alternatives — variety is the product. And where the task has one correct output and you can check it — code with tests, arithmetic with a verifier — the strongest configuration is often neither extreme: sample several at moderate temperature and let the check decide.

• Top-p and top-k truncation in detail, and why applying them before temperature changes the behaviour so much.
• Best-of-n sampling with a verifier or reward model, and where the gains stop.
• Beam search, which attacks greedy decoding by keeping several partial sequences rather than by adding randomness.

Every term the collection defines is gathered in the glossary.