Scaling that overshoots
A Socratic walk-through of scaling that overshoots — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why does a system that adds capacity by itself swing between far too much and far too little?
An autoscaler is a simple promise: watch the load, add machines when it rises, remove them when it falls. Nothing about that sounds like it should produce a sawtooth. And yet the graph a team brings to a post-mortem is almost always a sawtooth — eighty instances, then twelve, then ninety, then fifteen — with the load underneath it wandering along quite gently.
So the interesting question is not why did it scale wrong. Each individual decision was correct given the number it saw. The question is why a sequence of locally correct decisions produces a wildly incorrect trajectory.
Reasoning it through
REASONING #Start by counting the delays. When demand rises at ten o'clock, when does a new instance actually serve a request? The metric has to be sampled, then aggregated over a window, then published; that is tens of seconds. The scaler evaluates on its own interval; more seconds. The instance is requested, provisioned, booted, its runtime warmed, its caches filled, its health check passed; that is often minutes. Add them and the gap between the load changed and the capacity changed is not instant. It is a lag.
Now the crucial move. During that entire lag, the signal the scaler is watching keeps saying still overloaded — because it is still overloaded, the help has not arrived. What does a scaler that reacts to every observation do with that? It scales again. And again. By the time the first batch of capacity lands, four more batches are already in flight, ordered against a shortage that the first batch has now cured.
Can you see the shape of it? The system is not responding to the load. It is responding to its own not-yet-arrived response. Overshoot is the arithmetic consequence.
Then the second half. With far too much capacity, the metric collapses — CPU at four percent — and the same logic runs in reverse, at the same eagerness, and it removes too much. The trough is not a separate bug. It is the identical mechanism with the sign flipped.
Control engineers have a name for the ingredients: a feedback loop with gain (how hard you react per unit of error) and lag (how long before your reaction shows up in the signal). Their result is unkind to intuition — past a certain point, increasing gain in a lagged loop does not make the system converge faster, it makes it oscillate. A slow, deliberate response is not timidity. It is the only response that is stable.
There is a third ingredient, and it is the one most often missed: which signal you feed the loop. CPU utilisation is a symptom that appears well after the cause. Requests have already been queuing for a while by the time processors are pinned. Choose a metric closer to the demand — queue depth, in-flight requests, messages waiting per consumer — and you have shortened the lag before touching any other setting.
And a hazard worth naming honestly: scaling can create the load it is reacting to. New instances often stampede a shared dependency at boot — opening connection pools, warming caches, hammering a config service — so the database slows, the metric worsens, and the scaler orders more. Whether a given system does this depends on what its startup path touches, so it is a thing to check rather than assume.
The analogy
THE ANALOGY #Think of a shower with a long pipe. You turn the tap; nothing changes for six seconds. So you turn it further. Then the water arrives scalding, and you crank it back hard — and six seconds later you are in ice water. The tap is working perfectly. Every adjustment you made was a sensible answer to the temperature at your skin. The oscillation lives entirely in the delay between the handle and the water, plus your willingness to keep adjusting during it.
the shower's delay is roughly the same in both directions, whereas capacity is deeply asymmetric — adding an instance takes minutes, removing one is instant — so the real system falls faster than it rises, which is why the troughs bite harder than the peaks and why the two directions need different rules rather than one mirrored one.
Clarifying the model
THE MODEL #If the disease is reacting during the lag, every real fix is a way of not doing that.
Wait after acting. Cooldowns, stabilisation windows, and instance warm-up periods all say the same thing: having ordered capacity, do not order more until the last order could plausibly have shown up. Kubernetes exposes this as a stabilisation window on the horizontal pod autoscaler; the cloud providers call it cooldown or warm-up. It is the single highest-value setting and it is routinely left at a default chosen for a faster-booting workload than yours.
Make the two directions different. Scale up quickly on a small excess; scale down slowly, and only on a sustained deficit. That asymmetry — hysteresis — deliberately introduces a dead band where nothing happens, which is exactly where a jittery loop needs to spend most of its time.
Stop treating the target as a line. A scaler chasing "exactly seventy percent" will always be moving, because the number is never exactly seventy. A band it must leave before anything happens converts noise into silence.
Shorten the lag itself. Pre-warmed pools, snapshotted images, lighter startup paths, and scheduled scaling ahead of a known daily peak all attack the delay rather than compensating for it. Predictive scaling is the same idea: if you act on the forecast rather than the measurement, you are no longer inside the loop.
The misconception worth correcting: overshoot is not a sign that the thresholds are wrong. Tuning the threshold moves where the oscillation centres; it does not remove it. What removes it is reducing gain, adding patience, or shortening lag — and a team that only ever edits the target number will chase the sawtooth around the graph forever.
A picture of it
THE PICTURE #How to readStart at the parallelogram, the incoming demand. Follow the diamond's yes branch down the ordinary path — order, provision, warm up, serve — and notice how long that leg is. Then take the labelled edge back up from the metric into the shaded node on the right: that is the loop closing on evidence gathered before the help arrived, and it feeds straight back into ordering. The second shaded node is the same loop running in reverse once the surplus lands.
What became clearer
WHAT CLEARED #The oscillation is not caused by a bad rule. It is caused by a correct rule applied faster than the system can answer — so the loop ends up measuring its own outstanding orders and mistaking them for unmet demand.
Where to go next
ONWARD #- Queue depth and in-flight requests as leading indicators, against CPU as a lagging one.
- Predictive and scheduled scaling: acting on a forecast to leave the feedback loop entirely.
- Why cold-start cost sets a floor on how responsive any autoscaler can safely be.
Key terms
TERMS #| Term | What it means |
|---|---|
| Control lag | the delay between a control action and its effect appearing in the measured signal. |
| Gain | how large a corrective action is taken per unit of measured error. |
| Hysteresis | using different thresholds for acting in each direction, creating a band where nothing happens. |
| Cooldown / stabilisation window | an enforced pause after a scaling action, during which further actions are suppressed or damped. |
| Warm-up period | the interval after an instance starts before its metrics are trusted or it is counted as serving. |
Every term the collection defines is gathered in the glossary.