Keeping the history and the snapshot in one file

How can one log hold both every change ever made and just the current state?

A log is a record of everything that happened, in order, forever. A table is a record of how things are right now. They look like opposites: one grows without bound and remembers, the other stays small and forgets.

Yet a compacted log claims to be both at once, and a new reader can rebuild an entire table from it without replaying years of traffic. That should sound impossible before it sounds clever. Where would the old changes go, and how does throwing them away leave the current state intact?

Start with what a table actually is. Ask what you would have to keep, from a stream of updates, to answer "what is the value of key K?" for every K. Not the whole stream — only the last record for each key. Everything before it was overwritten by something later, and no query about the present can distinguish a log that kept those earlier records from one that did not.

So the two things are not opposites at all. A table is a lossy projection of a log, and the loss is precisely the superseded records. That gives the mechanism: walk the log, and for each key discard everything except the most recent entry. The result is smaller than the original by however much rewriting happened, and it is still a log — same append-only file, same offsets, readable by the same consumer.

But now push on the awkward part. What about deletion? If a key's absence is expressed by simply having no record, then removing the last record for a deleted key would leave a reader that already has the old value with no way to learn it was removed. So deletion needs its own record — a key with a null value, a tombstone — which is kept for a grace period so live readers see it, and only then removed. That grace window is the price of expressing absence in a medium that only knows how to append.

Here is the observation that answers the original question, and it is the one that usually clicks late. Compaction does not apply to the whole file. The segment currently being written is never compacted, and older segments are only cleaned when enough of them has gone stale. So at any moment the log has two regions: a recent tail that is complete, unedited history, and an older head that has been reduced toward a snapshot. It is not both things everywhere. It is history where history is still fresh and summary where it is not.

Which explains why the two readers see different things without contradiction. A consumer that has been attached all along reads every record as it is appended, so it sees the full sequence of changes. A consumer starting from the beginning today reads a head that has been thinned, and what it is promised is narrower but sufficient: at least the final value of every key that has one. It gets the present state, not the past.

Two honest limits. Compaction is opportunistic, not scheduled: a background cleaner picks logs by how much of them is stale — Kafka's default threshold is half — so a superseded record can survive for a long, unbounded time. If you need an old value gone by a deadline, compaction alone will not give you that; you need the explicit lag settings, and even then you are bounding when cleaning may begin. And second, this only works for keyed streams whose semantics are "latest wins". A stream of measurements, or of events that accumulate rather than replace, has no superseded records to remove, so compaction has nothing to reclaim and would be the wrong retention policy entirely.

The misconception to name is that compaction is a retention policy in the same sense as "delete after seven days". Time-based retention discards by age, which can delete the only record of a key that has not changed in a year — and that key is the most stable, most important part of your state. Compaction discards by redundancy, which is why the two are answers to different questions and why choosing the wrong one silently destroys data rather than merely storing too much.

The refinement that makes this powerful: because a compacted log is bounded by the number of keys rather than the number of events, restoring state from it is bounded work. A stream processor that keeps a local store can write every change to a compacted topic and, after any crash on any machine, rebuild the store by replaying it — and the replay cost tracks the size of the state, not the age of the system. That is the property that makes the pattern practical rather than merely elegant.

And one thing that does not survive: order across keys. Compaction preserves the relative order of the records it keeps, but the gaps between them are real. You cannot use a compacted head to reason about what happened between two events, only about where things ended up.

• How stream-table duality is used to join a changing dimension against a stream of events without a database in the middle.
• Why "delete this person's data" is hard on a compacted topic, and what the minimum and maximum compaction lag settings can and cannot promise.

Every term the collection defines is gathered in the glossary.