Prompt sensitivity
A Socratic walk-through of prompt sensitivity — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why does rewording a request without changing its meaning change the answer that comes back?
You ask a question, get an unsatisfying answer, rephrase it in a way you consider identical in meaning, and get a noticeably better one. Nothing about the world changed. Nothing about the model changed. Only the wording changed.
The tempting reading is that the model "understood you better the second time", which quietly assumes there is a stable understanding in there that your first phrasing failed to unlock. Before accepting that, it is worth asking a blunter question: on what grounds did we expect two paraphrases to produce the same output at all? What in the machinery guarantees that meaning, rather than wording, is what it responds to?
Reasoning it through
REASONING #Follow the input in. Your words become tokens, the tokens become vectors, and everything downstream is arithmetic on those vectors. Two paraphrases are two different token sequences, so they produce two different starting points and two different trajectories through the network. The output is a probability distribution over the next token, computed from wherever that trajectory arrives.
So the honest default expectation is the opposite of the one we started with. Identical outputs would require the model to map both phrasings to the same place. It does not do that exactly — it does something weaker and far more useful, which is to map them nearby. Most of the time nearby is enough, which is precisely why paraphrase robustness feels like the norm and sensitivity feels like a glitch. But near is not the same as equal, and when the distribution over next tokens is nearly balanced between two continuations, a small nudge decides it. Ask yourself what a small difference does at a near-tie: it flips the outcome completely. That is the whole shape of the phenomenon — smooth almost everywhere, discontinuous at the boundaries.
Now notice that not all rewordings are equal, and this is where measurement has been genuinely surprising. Changing the order of few-shot examples, holding their content fixed, was shown by Lu and colleagues in 2021 to move accuracy across an enormous range on the same task — from near state of the art to near chance, depending purely on permutation. Zhao and colleagues, the same year, traced part of this to identifiable biases: a pull toward whichever label appeared most often in the examples, toward the label seen most recently, and toward tokens that are common in general. Sclar and colleagues, in 2023, went further and varied only cosmetic formatting — separators, spacing, casing, whether a field is written Answer: or A: — and found accuracy spreads of tens of points across formats that no human reader would consider different prompts.
Set beside that, the effects people talk about most are often the smaller ones. Politeness, flattery, and threats have been studied and the results are mixed and modest, not the large reliable levers folklore suggests. Meanwhile a phrasing that changes what the model does rather than how it is addressed — asking for reasoning before the answer, in the manner Kojima and colleagues showed in 2022 with a step-by-step instruction — can change results substantially on tasks where the extra tokens actually do work. That distinction is a useful sorting rule: a rewording that changes the computation matters more than a rewording that changes the tone.
Two mechanisms sit underneath all this and are worth separating. One is genuine sensitivity: different tokens, different trajectory, different distribution. The other is sampling — if you are not at temperature zero, two runs of the same prompt can differ, so some of what looks like sensitivity to your rewrite is noise you would have got anyway. Anyone comparing prompts seriously has to run each several times, or the comparison measures the dice.
And an honest limit: why a particular rewording helps a particular task is usually not known. The empirical fact of high variance is solid and repeatedly measured. The story you tell about why "as an expert" helped is, in most cases, a story.
The analogy
THE ANALOGY #Think of a ball resting on a landscape of hills and valleys. Nudge it anywhere on a broad slope and it rolls to the same valley floor; the nudge is absorbed. Nudge it while it is balanced on a ridge between two valleys, and the same tiny push sends it somewhere else entirely. Your rewordings are all the same size; what varies is whether the question you asked left the model on a slope or on a ridge.
a ball on a ridge visibly wobbles before it falls, whereas the model gives no signal that it was near a boundary — both answers arrive with the same fluent confidence, which is why the instability is invisible until you test for it.
Clarifying the model
THE MODEL #Three refinements. First, sensitivity is not a sign the model failed to understand you. It is what conditioning on tokens rather than on meanings looks like from the outside, and it is present in every model of this kind, including ones that are otherwise very capable.
Second, it is a measurement problem before it is a prompting problem. A benchmark score for a single phrasing reports one sample from a distribution, and comparing two models on one prompt each can rank them by their luck with formatting. This is why careful evaluations now report spread across paraphrases and formats rather than a single number.
Third, instruction tuning genuinely reduces the effect — that is much of what it is for — and it has not eliminated it. So the practical posture is neither superstition nor complacency: prefer prompts that specify the task and the output shape explicitly, keep the format fixed when comparing anything, sample more than once, and treat a phrasing that only works sometimes as evidence the task is near a boundary rather than as a magic word you have found.
A picture of it
THE PICTURE #How to readread left to right as how much of the prompt text you actually altered, and bottom to top as how much the answer tends to move. The interesting region is the upper left: edits almost too small to notice that nonetheless swing results, which is where the reported evidence on example order and formatting sits. The placements are qualitative summaries of the studies named above, not measurements of any one task — treat the quadrant a point falls in as the claim, not its coordinates.
What became clearer
WHAT CLEARED #The surprise was in the expectation, not the behaviour. A model conditions on the exact tokens you supplied, so different wording is genuinely different input; that it usually gives the same answer anyway is the achievement, and the exceptions cluster wherever the decision was close. What the measurements add is which knobs are big: order and formatting move results far more than tone does, and any comparison that varies the wording without holding the format fixed is partly measuring noise.
Where to go next
ONWARD #- Why calibration methods that correct for majority-label and recency bias recover much of the lost accuracy.
- How to evaluate a prompt honestly: multiple paraphrases, multiple samples, reported spread.
- Whether models that reason before answering are more robust to phrasing, or merely shift the sensitivity elsewhere.
Key terms
TERMS #| Term | What it means |
|---|---|
| Few-shot prompting | supplying worked examples in the prompt so the model infers the task from them. |
| Temperature | the sampling parameter controlling how sharply the next token is drawn from the model's distribution; above zero, repeated runs differ. |
| Instruction tuning | fine-tuning on demonstrations of following instructions, which improves but does not remove phrasing sensitivity. |
| Majority-label bias | the pull toward whichever answer label appeared most often among the supplied examples, regardless of the question. |
Every term the collection defines is gathered in the glossary.