Next-token prediction

Why does a system trained only to guess the next word end up able to summarize, translate, and write code?

The training objective is described in one sentence: given some text, put a probability on every possible next token, compare that guess with the token that actually came next, and nudge the weights to make the true one likelier. Repeat a few trillion times. Nothing in that sentence mentions summarizing, translating, or writing code.

So one of two things must be true. Either the description is hiding an extra ingredient, or "guess the next word" is a far more demanding task than it sounds. Which would you bet on? I want to argue the second is mostly right — and then be honest about the extra ingredient, because there genuinely is one, and it is not the part people usually suspect.

Try the task yourself. I hand you a page of text with the last word removed and ask you to name it. Easy, you say. But notice I never said which page.

Suppose the page ends "the French word for bread is". To score well you must know French. Suppose it ends with the last line of a murder mystery: "and the killer was". Now you must have followed a plot across two hundred pages and reasoned about who had motive and opportunity. Suppose it is a Python file where a docstring says the function returns the second-largest element, and the next line is the first line of the body. Now you must be able to write code that does that. Suppose it is a forum thread: a long question, then the accepted answer. To predict the answer's first words you must know the answer.

That is the whole trick, and it is worth pausing on it. Nobody assembled a curriculum of translation, summarization, and coding tasks. Those tasks are simply what continuing human text looks like, over and over, in a corpus large enough to contain bilingual pages, papers with abstracts, questions with answers, and code with comments. The objective did not have to be told about them; it could not be satisfied without them.

Now ask what the pressure feels like from the inside. The loss is computed on every single token, so the model is not sitting one exam — it is sitting billions of one-question exams, and every fact, every grammatical agreement, every closing bracket is worth a fraction of a point. Ask yourself: with that much text, which is cheaper — storing the surface statistics of the corpus, or building a compressed internal model of the things the corpus is about? Memorization does not scale here; there is far more text than there is capacity to store it verbatim. Something more general has to be learned to keep the loss going down. That framing — that good prediction forces good compression, and good compression looks a lot like understanding — is genuinely suggestive rather than proven, and it is worth holding loosely.

Here is the ingredient the one-sentence description leaves out. A model trained only on next-token prediction is what the field calls a base model, and a base model does not behave like an assistant. Ask one a question and a very reasonable continuation is another question, or a list of similar questions, because that is what a page of questions usually contains. Willingness to follow an instruction, to stop when the answer is done, to refuse, to adopt a helpful register — that comes from later stages: supervised fine-tuning on demonstrations of instruction-following, then preference-based tuning such as RLHF or DPO. So the honest form of the claim is that next-token prediction supplies the capability, and the later stages supply the disposition to use it when asked.

One more caution, because it is often stated too confidently. The word usually attached to all this is "emergence": abilities that appear absent at small scale and present at large. Schaeffer and colleagues argued in 2023 that many of these sharp jumps are partly an artefact of how we measure — score a task with exact-match accuracy and improvement looks like a cliff, score the same runs with a metric that gives partial credit and the same curve is often smooth. The capabilities are real; the suddenness may be a property of the ruler.

A few refinements that connect the steps. First, it is a token, not a word — a subword fragment — which is why spelling and arithmetic behave oddly, but that is a separate story. Second, the output is a distribution over the whole vocabulary, not a single guess; how sharply you sample from it is the temperature setting, which is why the same prompt can give different answers.

Third, and most often misstated: "one token at a time" is a description of the output, not of the computation. It is tempting to conclude the model can never plan ahead. That conclusion is not safe — interpretability work at Anthropic in 2025 found evidence of a model choosing a rhyme word before writing the line that leads to it, and then constructing the line to land there. How general that is remains an open research question, but the naive picture of pure word-by-word improvisation is already too simple.

And keep the two stages separate in your head. When an assistant follows your instruction, next-token prediction is why it can; fine-tuning is why it does.

• Why the same objective produces confident fabrication, and whether that is a bug in the objective or in how we sample from it.
• What "scaling laws" actually predict, and what they conspicuously do not.
• How supervised fine-tuning and preference tuning change a base model's behaviour without teaching it new facts.

Every term the collection defines is gathered in the glossary.