Hot partitions

Why does data split evenly across a hundred machines still overwhelm exactly one of them?

You have hashed your keys, checked the distribution, and confirmed it: each of a hundred partitions holds within a per cent or two of the same number of rows. By any storage measure the split is perfect.

Then one partition saturates. Its latency climbs, it starts rejecting writes, and the other ninety-nine are nearly idle. The obvious response is that the hash must be bad — but the hash is fine, and rebalancing will not help. The fault is an assumption underneath the whole exercise, worth naming precisely because most people do not realise they are making it.

The assumption is that rows and requests are the same currency. You balanced rows; the machine spends its capacity on requests. The two match only if every row is accessed about as often as every other — and when is that true?

Think about real access patterns. One celebrity account among a million. One product on the front page on launch day. One tenant twenty times the median size. One sensor reporting every second while the rest report hourly. In each case the row count is unremarkable and the request count is not. Access frequency in human-facing systems is strongly skewed, often roughly power-law shaped, so a small fraction of keys draws a large fraction of the traffic.

Now the crucial structural fact: hashing preserves that skew instead of spreading it. Hashing spreads keys uniformly, which is what you want for storage. But every request for a given key deterministically lands on the same partition — it must, or the partition could not answer it. So a key drawing five per cent of all traffic sends five per cent to one machine, however many machines exist. Adding partitions does not divide that load; it makes the imbalance more visible, because the others get quieter while the hot one does not.

That produces a stark conclusion: a single key is an atomic unit of load. Its ceiling is one partition's capacity, and no horizontal scaling raises it. Total capacity scales; the hottest key's does not.

There is a second, more insidious route. Consider a key that looks perfectly uniform: a timestamp, or a monotonically increasing ID. Rows spread evenly across the key space over time — but at any given moment every write goes to the highest key. In a range-partitioned store that is one partition, always the last, the earlier ones read-only history. This is the classic time-series hotspot, worse than the celebrity case because the hot partition keeps moving, so you cannot even give it dedicated capacity.

Two distinct causes, then, with the same symptom. For the monotonic one, break the ordering — prefix the key with a hash or bucket number so consecutive writes land in different partitions. You pay by losing cheap range scans over recent data, often the reason the key was ordered at all.

For the skewed case the standard technique is key splitting, or salting. Replace the hot key celebrity with celebrity#0 through celebrity#15, chosen at random on write, spreading its load across sixteen partitions; reads must then query all sixteen and merge. Note the trade: you convert a single-partition query into a fan-out in order to convert a single-partition load into a distributed one. It is worth doing only for the few keys that need it — detecting hot keys at runtime and splitting them adaptively is what DynamoDB and Bigtable do internally.

For read-heavy skew there is a simpler answer that is often right: cache. A key hot enough to overwhelm a partition is by definition requested often enough that caching works extremely well — though it does nothing for write skew, the harder half. And an honest limitation: nothing here helps a key that must be written frequently and read consistently, such as a counter on a viral post. That case has only trade-offs — batch the writes and accept staleness, shard the counter and accept that reading means summing shards, or serialise and accept the ceiling.

Three refinements.

First, this is not a flaw in hashing. Hashing does its job perfectly; the job is simply not the one you needed. The mismatch is between what was balanced (rows, at design time) and what is consumed (requests, at run time), and no hash can fix that, because it cannot see the future access pattern.

Second, beware "we tested it and the load was even". Hotspots are frequently emergent — the key that is hot today was ordinary last month, because a product launched or an account grew. Skew is a property of the workload, and workloads change without warning. That makes this a monitoring problem as much as a design one: the useful instrument is per-partition request rate, not per-partition size, and the two are routinely conflated on dashboards.

Third, a correction worth stating: the hot partition is usually not "overloaded" in the sense of holding too much data. It is throttled, or its queue has outgrown its latency budget. And because most clients retry on timeout, it receives more traffic exactly as it becomes least able to serve it — so a small skew can tip into a full stall much faster than the underlying imbalance suggests.

• How adaptive systems detect a hot key at runtime and split it without application changes.
• Why retry storms turn a mild hotspot into an outage, and what backoff and load shedding do about it.

Every term the collection defines is gathered in the glossary.