Rate limiting

Why does a service protect itself by turning away work it could have handled?

A rate limiter refuses requests. Not requests the service cannot handle — it refuses at a threshold set below what the machines can do, and it refuses in the moment, cheerfully, with a 429 status code and no attempt at heroics.

The instinct is that this is leaving money on the table. If the servers could have served that request, serving it is free revenue. So what is the limiter protecting, and why is the safe threshold below the capable one?

Ask what actually happens to a service driven at the edge of its capacity. Not past it — at it.

The relevant fact comes from queueing theory and it is worth stating plainly: as utilisation approaches one hundred percent, waiting time does not rise proportionally, it rises towards infinity. At fifty percent utilisation a queue is barely a queue. At ninety percent the average wait has grown severalfold. At ninety-nine it is unbounded in the model, bounded in practice only by the point at which requests start timing out. So a service running at "full capacity" is not efficiently utilised; it is one small burst away from a latency cliff.

Now ask what a request that times out is worth. The client has given up. The server, however, usually keeps working on it — and then produces an answer nobody is waiting for. That work consumed a thread, a database query, a downstream call, all spent to produce nothing. So beyond a certain load, additional accepted requests do not merely fail; they convert capacity into waste, reducing the number of requests that succeed. This is the point most people have to see twice before believing it: accepting more work can produce less completed work.

Then remember what a timed-out client does next. It retries. And you have arrived at the retry storm — failure generating load, load generating failure, a loop that persists after whatever triggered it has gone. Rate limiting is the intervention that keeps the system on the safe side of the cliff, and it is much cheaper to stay there than to climb back.

So now the refusal looks different. Refusing a request costs almost nothing — one comparison and a small response. Accepting a request you cannot finish costs a full unit of work and also fails. Given a choice between a fast, honest no and a slow, expensive no, the fast one leaves more capacity to serve the requests you did accept. The limiter is not turning away work it could have handled; it is refusing to enter the region where handling anything gets worse.

How does the mechanism work? The classic shape is a token bucket. Tokens are added at the rate you want to sustain, up to a maximum the bucket can hold. Every request removes one; a request that finds the bucket empty is refused. Notice that this gives you two dials rather than one: the fill rate sets the long-run average, and the bucket size sets how large a burst you will absorb. That is precisely what you want, because real traffic is bursty and a limiter that permitted no burst at all would refuse ordinary customers behaving normally. The leaky bucket variant enforces a smooth outflow instead.

Then the question of whose rate is being limited — and here the purpose splits in two, which is worth separating carefully. A per-client limit exists for fairness: it stops one noisy customer consuming a resource everyone shares, and it makes each customer's cost predictable. A global limit exists for self-protection: it keeps total load below the cliff whatever the mix. They are different goals and can conflict, and a system usually needs both, since per-client limits alone will not save you when every client is busy at once.

A third mechanism is often conflated with limiting. Load shedding is dynamic: the service watches its own health — queue depth, latency, CPU — and drops the least valuable work when it is in trouble, handling the case a fixed number cannot, namely that real capacity moves with request mix and dependency health. Shedding is prioritised: reject bulk and retryable traffic first, protect the interactive paths.

One protocol detail matters more than it looks. A refusal should say when to come back: HTTP's 429 status, defined in RFC 6585, is meant to carry a Retry-After header. Without it clients back off blindly, and enough of them backing off by the same amount return in a synchronised wave — the refusal then organises the very burst it was defending against.

Some refinements.

The threshold is not a hardware number. It is the load at which latency and error rate stay inside what you promised, always lower than what the machines can absorb, and it moves as caches warm or dependencies slow — which is the argument for pairing a static limit with dynamic shedding rather than choosing between them.

Rate limiting is also not primarily an anti-abuse feature: the usual trigger is a well-behaved customer's batch job, or your own retry logic, so limits placed only at the public edge are looking the wrong way.

Finally, refusing is a decision about which work to lose, not about whether to lose any. Above the cliff, some requests will fail regardless; the only choice is whether they fail early and cheaply, chosen by you, or late and expensively, chosen at random by timeouts. Seen that way, a limiter is a prioritisation tool wearing a capacity tool's costume — and this is exactly why the shedding rules deserve as much thought as the number.

• Retry storms, and how a limiter's refusal interacts with a client's backoff policy.
• Adaptive concurrency limits, which infer the safe level from observed latency instead of a configured number.

Every term the collection defines is gathered in the glossary.