Talked out of its own rules

Why can a system trained to refuse be persuaded to agree?

Nobody argues a database out of a foreign key constraint. You can be as persuasive as you like; the write fails. Yet a language model that has been carefully trained to decline certain requests can, with the right framing, be brought to comply — not by finding a bug, but by talking to it.

That asymmetry is the thing worth understanding. What kind of constraint is a trained refusal, such that it has a rhetorical surface at all?

Start with where the refusal physically lives. When a model declines, no separate component inspected the request and returned a verdict. The refusal was generated — produced by the same machinery, from the same weights, in the same pass, as any other sentence. Training has made "I cannot help with that" a highly probable continuation for certain inputs. That is the whole mechanism.

So what have we actually built? Not a rule, but a strong statistical tendency, conditioned on the input. And once you say it that way, a question follows naturally: how far does that conditioning extend? A tendency learned on one distribution of inputs has to generalise to reach an input it never saw. Do we have any reason to expect it generalises as far as the model's capabilities do?

We have reason to expect the opposite, and this is one of the two mechanisms the research literature identifies. Safety training is a comparatively thin layer applied over an enormous pretraining corpus. The capability to produce some piece of content came from that corpus; the disposition to refuse came from the thin layer. Move the request into a region the thin layer never covered — an unusual encoding, a lower-resource language, a nested fiction, a translation task — and the capability is still there while the refusal behaviour has nothing to interpolate from. Nothing was overridden. The refusal simply was not triggered, because this input does not look, to the model, like the inputs refusal was taught on.

The second mechanism is different and worth separating. Refusing is not the model's only trained objective. It has also been shaped, very strongly, to be helpful, to follow instructions, to continue text coherently, to stay in a role it has been given. An attacker who constructs a prompt in which those objectives point away from refusal is not defeating safety training; they are setting it against the rest of the training. Instructing the reply to begin with a particular affirmative phrase, or forbidding refusal language, exploits exactly this — once a compliant opening is on the page, the coherence pressure to continue it is real.

Now notice what both mechanisms share. Neither involves a rule being broken. There is no boundary to cross, because there is no boundary — there is one continuous function from inputs to output distributions, and safety training reshaped part of its landscape. An attacker is searching for a nearby input that lands somewhere the reshaping did not reach.

Does that make the search easy? Not always, but there are worrying signs about its cost. Attacks can be found automatically by optimisation rather than by human wit, and suffixes discovered against one model have transferred to others. Filling a long context with many fabricated example exchanges raises success rates as the number of examples grows. And the deeper structural problem is that there is no privileged channel: a system prompt, a user message, and a retrieved document all arrive as tokens, distinguished only by conventions the model has learned to respect — and a learned convention is exactly the kind of thing this whole discussion says can be worked around.

Is this solvable? Honestly, it is not settled. Training has improved substantially and continues to; no one has shown a training method that closes the gap in general, and the current practical answer is layers rather than a fix.

The refinement to hold on to is the distinction the whole subject turns on: a trained constraint is a disposition inside the generator, and an enforced constraint is a check outside it. Only the second is immune to argument, and it is immune for an unglamorous reason — it does not read the argument as an instruction. A tool that will not execute a shell command, an account that cannot see a record, a classifier that inspects the output after the fact: none of these can be reasoned with, because none of them process reasoning.

The misconception worth correcting is that a jailbreak is an exploit in the sense of a buffer overflow — some flaw that a careful enough implementation removes. It is better understood as a distribution problem. The behaviour was learned from examples, the input space is unbounded, and the attacker gets to choose the input. That framing explains why patching a specific prompt achieves so little: the patched prompt was never the vulnerability, it was one sample from a region.

It also explains where effort actually pays. Do not ask the model to be the last line of defence for anything that matters. Give it the least authority that lets it do its job, put real checks on what it can touch, and treat safety training as one useful layer that raises the cost of an attack rather than as a control that ends one.

• How competing objectives and mismatched generalisation differ as attack recipes, and which defences address each.
• Why adversarial suffixes found against one model often transfer to another.
• What least-privilege actually looks like for a model with tools, and how to scope its authority.

Every term the collection defines is gathered in the glossary.