Copying something you can only question

How do you reconstruct a system you are never allowed to look inside?

A company spends a fortune training a model and then does the only commercially sensible thing: it hides the weights and sells access to the answers. You may ask it anything; you may not see it. That feels like a safe trade, because the valuable object never leaves the building.

But consider what you are being handed instead. Every query returns a point on the function the model computes. Ask enough questions and you are accumulating a description of that function. So the question worth sitting with is not whether answers leak information about the thing that produced them — they must — but exactly what they leak, and where that leakage stops. Those two halves get conflated constantly, and the conflation is where both the panic and the complacency come from.

Start with the simplest case, because it is the one where the answer is exact. Suppose the hidden model is a logistic regression over ten features, and the API returns a confidence score rather than a bare label. Each query then gives you one equation in the model's parameters. Eleven well-chosen queries give you eleven equations, and you solve them. Tramer and colleagues demonstrated exactly this at USENIX Security in 2016 against commercial prediction APIs, recovering the parameters of logistic regressions and decision trees outright. Nothing statistical about it — it is algebra, and the leak came from returning confidences rather than labels.

Now scale that up and watch the guarantee weaken. For a deep network there is no small system of equations to solve, so what does querying buy you instead? Training data. Each query-and-answer pair is a labelled example produced by the target itself, and a fresh model trained on enough of them will approximate the target's behaviour. Notice what has changed: you are no longer recovering the model, you are training a substitute on its outputs — distillation, done without permission.

That distinction has consequences worth being precise about, and the literature separates them. Fidelity is how closely the copy agrees with the original, including on the original's mistakes. Accuracy is how often the copy is right. A distilled substitute can reach high accuracy while diverging from the target on the awkward inputs, and if your reason for copying is to find adversarial examples that transfer, fidelity is the property you actually need.

Then ask the sharper question: is exact weight recovery even a coherent goal? For a neural network, no. Permute the neurons in a hidden layer and rescale a pair of weights and you get a numerically different network that computes exactly the same function. Behaviour cannot distinguish them, because behaviour is all you ever see. Anything recovered from queries is recovered up to those symmetries at best.

Which makes the 2024 result from Carlini and colleagues instructive rather than contradictory. They showed that the final embedding projection layer of a production language model could be recovered — along with the hidden dimension — from ordinary API access, for a modest query cost. But read the claim carefully: it is the last layer, and it is recovered up to an affine transformation, because that is the resolution the outputs allow. The rest of the network stayed hidden. What made even this possible was an interface returning fine-grained logit information, and the mitigation was to restrict that interface.

So collect the pattern. The richer the output, the more each query tells you: full logit vectors leak the most, then ranked probabilities, then a top-1 label with a score, then a bare label. And what does not come out? Not the training data — that is memorisation extraction, a different attack with a different mechanism. Not the architecture beyond what the outputs constrain. Not the interior of the network. What you get is the input-output map, sampled where you chose to sample it, which means the copy is faithful on the distribution you queried and unreliable off it.

Three clarifications.

First, the common overstatement: extraction does not "steal the model" in the sense of recovering the weights. Outside small, structured models it produces a functional imitation whose quality is bounded by how well the query distribution covered the input space.

Second, the common understatement: an imitation is often enough for the attacker's purpose. It removes the per-query fee, it can be probed offline without rate limits, and it serves as a local stand-in for crafting attacks meant to transfer back to the original.

Third, the defences follow directly from the reasoning and each has a cost. Return less — labels rather than confidences, rounded scores rather than exact ones — and you lose utility for honest callers. Rate-limit and monitor for query patterns that look like systematic coverage rather than use, and you risk false positives. Watermark the outputs, which does not prevent copying but supports a later claim of ownership. There is no defence that keeps the interface fully informative and fully private, because the information is the interface.

• Distillation as the legitimate twin of this attack, and why the line between them is contractual rather than technical.
• Membership inference and memorisation extraction: what queries reveal about the training data rather than the model.
• Watermarking model outputs, and how robust such marks are to paraphrase.

Every term the collection defines is gathered in the glossary.