Data leakage paths

How does private information escape a system that nobody attacked?

Most breach stories have a villain. But a large share of real disclosures have none: no intrusion, no stolen credential, no exploited flaw. A support engineer opens a ticket and finds a customer's full record pasted into it. A search index quietly returns a document its reader was never entitled to see. A staging database, refreshed from production for testing, sits on a laptop.

If nobody attacked, what did the leaking? I want to suggest the answer is architectural rather than moral — and that the way to find it is to stop thinking about who might take the data and start thinking about where copies of it exist.

Here is the question I would put first. When you grant someone permission to read a record, where does that permission live?

In one place: the system that owns the record. The check happens at the database, or at the API in front of it. Now trace what happens to the same bytes over the next few seconds. They go into a log line for debugging. They pass through a cache. They appear in an error report when something downstream fails. They get embedded into a vector and stored in an index. They are copied into last night's backup. They are counted into an analytics event. They may be forwarded to a third-party service that does something useful with them.

How many of those copies carried the permission with them?

Almost none, and that is the whole mechanism. Access control is a property of the original store, but the data is a travelling thing. Every copy is created for some function — debugging, speed, analysis, resilience — and each function imposes its own structure, with its own access rules, its own retention, and usually its own much larger audience. Nobody decided that the log aggregator should be readable by every engineer; it simply is, because that is what a log aggregator is for.

So a leak, in this reading, is not an event. It is a pre-existing arrangement waiting for someone to notice it. The record was already reachable through a path that never asked the question the origin would have asked.

Now let us apply that to the case that has become the most common new source of these paths: a retrieval-augmented assistant. Documents are collected, chunked, embedded and stored in an index so the assistant can find relevant material. What happened to the permissions on those documents? If the index does not carry each chunk's access list and filter retrieval by the asking user, then a system whose purpose is to surface anything relevant will do exactly that, across every boundary the original file system enforced. The assistant did not break in; it was built to fetch, and nothing told it whom each fragment belonged to.

There is a subtler variant worth naming honestly. Even with correct filtering, a system can leak through what it reveals about data it will not show: whether a document exists, how many results matched, how long a request took. These side channels are real and are usually judged an acceptable residual risk rather than fully closed — an honest position, not a solved one.

The reframing that makes this tractable is to think in terms of egress paths rather than defences. Ask of any sensitive field: what are all the ways a byte of this can end up somewhere a person can read it? The answer usually falls into a small number of families.

Observability — logs, traces, metrics, crash dumps, stack traces containing request bodies. This is the most common accidental path, because logging is added under time pressure and a log statement is not thought of as a data flow.

Derived stores — caches, search indexes, vector indexes, materialised views, analytics warehouses. Each one is a copy whose access model was designed for its own function, not inherited from the source.

Environments — production data copied into staging, test fixtures, and developer machines, where controls are deliberately weaker.

Model surfaces — fine-tuning on user content, which can be regurgitated later; shared context or cache between tenants; and a system prompt or tool description that quietly contains something confidential.

Human channels — support tooling with broad read access, screenshots pasted into tickets, exports to spreadsheets, screen shares.

Third parties — every SDK, subprocessor and vendor that legitimately receives a slice of the data, each with its own version of all of the above.

Two misconceptions worth correcting. First, encryption addresses very few of these: nearly every path listed here handles the data after decryption, in the ordinary course of doing its job. Second, "we have access control" is a statement about one node in a graph, and the graph is what matters — the useful audit is not "is the database locked" but "for this field, enumerate the copies".

The design principles that follow are unglamorous. Minimise what you collect, since a field never stored has no paths. Redact where data enters a derived store rather than trusting each consumer. Propagate the source permissions into every index built from permissioned material, and filter at query time by the asking identity. Set retention on the copies, not only on the original.

• Data flow mapping in practice, and how to keep the map current when copies are created by automated pipelines.
• Permission-aware retrieval for assistants: carrying access control lists through chunking and embedding, and filtering at query time.
• Memorisation in trained models — what can and cannot be recovered from the weights.

Every term the collection defines is gathered in the glossary.