Few-shot prompting
A Socratic walk-through of few-shot prompting — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why does showing a model three worked examples work better than describing the rule?
Suppose you want a model to pull dates out of messy sentences. You could write a paragraph explaining what counts as a date, how to normalise it, what to do with a range, what to do when there is none. Or you could paste three sentences with their answers underneath and stop.
The second usually works better, and the puzzle is why. The paragraph contains more information — it names cases the three examples never touch. So what is it that three examples give you that a careful description withholds?
Reasoning it through
REASONING #Start with what a rule actually is. A rule is a compression: it names a category and trusts the reader to unpack it. "Normalise the date" assumes a shared idea of normal. You know what you meant. The model has to guess which of the many things that phrase could mean is the one you had in mind — and the phrase is compatible with all of them.
An example does not compress. It shows an input and the exact output you want beside it. Every choice you were vague about is now settled in the only way that matters: demonstrated. The order of the fields, the capitalisation, whether there is a trailing period, what an empty answer looks like. Ask yourself how many sentences of prose it would take to pin down all of that. Then notice you spent none, because the example carried it for free.
Now push further. Where does the ambiguity in a rule live? Mostly at the edges — the cases you did not think to mention. Here examples do something subtler than illustrate: they mark out a region. Three examples of a certain kind of input suggest a family of inputs, and the model's continuation is shaped toward that family. If your three examples are all one-line customer emails, you have communicated "this is the sort of thing you will see" without saying it.
And the mechanism underneath? The model was trained to continue text. Your examples are a piece of text with an evident pattern: input, output, input, output. Continuing that text is doing the task. So a prompt with examples is not instructing the model at all — it is starting a document whose most natural continuation happens to be the answer you want. That is why the pattern is called in-context learning: nothing in the model's weights changed. The examples are working entirely as context.
Here is where I have to be careful, because the obvious inference from that story is wrong, or at least contested. You would assume the model reads the examples to infer the input-to-output mapping. Min and colleagues tested this in 2022 by randomly shuffling the labels in the demonstrations — pairing inputs with wrong answers — and found that on a range of classification tasks performance dropped far less than expected. What mattered more was that the examples showed the right format, the right label space, and the right distribution of inputs. Later work has complicated that picture again, with evidence that stronger models do use the mapping, and more so with more examples. So treat "it teaches the rule by induction" as unsettled. What is well established is the humbler part: examples set format, scope, and register with a precision prose does not reach.
One more observation, and it is the one that decides when not to use examples. Some rules genuinely are simple to state and hard to demonstrate — "always answer in fewer than fifty words", "never mention a competitor". Three examples that happen to be short do not communicate a limit; a sentence does. So the choice is not examples-versus-description as a matter of taste. It follows from whether the thing you want is easier to say or easier to show.
The analogy
THE ANALOGY #Think of teaching someone to fold a fitted sheet. You can describe it: tuck the corners into each other, align the elastic seam, fold in thirds. They will do it wrong. Fold one in front of them and they will do it nearly right, because your hands settled a hundred small questions — which corner first, how much tension, what "align" looks like — that your sentences left open, and settled them without either of you noticing the questions existed.
the learner watching your hands builds a lasting skill, whereas the model retains nothing at all once the prompt ends — next conversation, the demonstration must be given again, because the examples changed only the context, never the model.
Clarifying the model
THE MODEL #Two refinements connect the pieces.
The first is that examples and descriptions are not rivals; they are good at different things. A description states a constraint — a boundary the output must respect, especially a negative one. An example conveys a form — what a good answer looks and reads like. The strongest prompts usually carry both, a short rule for the boundaries and two or three examples for the shape, which is why the practical advice is rarely "drop the instructions".
The second is a caution about what examples smuggle in. Because they set the input distribution, three examples that are all easy cases quietly tell the model that the hard case is out of scope. Because they set format, three examples that all happen to be positive can bias the answer toward positive. Examples teach everything they contain, including their accidents — which is why choosing them carelessly is a real failure mode, and why an example with a mistake in it is worse than no example at all.
That corrects the common misconception that more examples are simply better. Adding examples helps sharply at first and then flattens; past that point you are spending context, and a badly chosen tenth example can do more harm than the first three did good.
A picture of it
THE PICTURE #How to readRead the three branches as three different jobs, not three benefits. The left-hand job belongs to prose and does not transfer to examples; the middle is what examples do that prose cannot; the third branch is the cost — everything the examples happen to have in common is taught alongside the lesson you intended, whether you meant it or not.
What became clearer
WHAT CLEARED #A worked example is not a shorter way of saying a rule. It is a different kind of communication: it settles by demonstration all the details a description leaves to interpretation, and it does so at no cost in words. What it cannot do is state a limit, and what it always does, whether you want it or not, is define the territory it came from. The skill is knowing which of your requirements are shapes and which are boundaries.
Where to go next
ONWARD #- Why performance from added examples flattens, and how that interacts with context length and cost.
- How example ordering and formatting change results, and what that implies about how brittle the effect is.
- The difference between in-context learning and fine-tuning, once the same demonstrations are used both ways.
Key terms
TERMS #| Term | What it means |
|---|---|
| Few-shot prompting | including a small number of input-output examples in the prompt rather than describing the task. |
| In-context learning | a model adapting its behaviour from the prompt alone, with no change to its weights. |
| Zero-shot | the same task attempted from a description with no examples at all. |
| Label space | the set of answers the examples show as permissible, which examples communicate implicitly. |
Every term the collection defines is gathered in the glossary.