Circuit breakers

Why does refusing to even attempt a request help a struggling system recover?

Here is a piece of advice that sounds like giving up: when a dependency starts failing, stop calling it. Not "call it more carefully" — stop, and return an error immediately without even trying.

It is worth resisting that for a moment. Surely the call might succeed? Surely refusing to try guarantees a failure that was only probable? And yet this pattern, named and popularised by Michael Nygard in Release It! in 2007, is standard in every serious distributed system. So what does the refusal buy, and who does it buy it for?

Start with the caller, and ask what a call to a sick service actually costs it.

When a dependency is healthy, a call costs a few milliseconds and a thread. When it is sick, the call usually does not fail fast — it hangs, until the timeout fires. So the same call now costs two seconds and holds a thread, a socket, and whatever memory the request context occupies for that whole time. If requests keep arriving at their normal rate while each one occupies resources a hundred times longer than usual, what happens to the caller?

It runs out. Threads are all parked waiting on a dependency that will not answer, and the caller stops serving everything — including the nine features that had nothing to do with the sick dependency. This is the first thing the breaker buys: one failing dependency stops being able to consume the caller's entire capacity. Failing in one millisecond instead of two thousand is not defeatism, it is the difference between a degraded feature and a dead process.

Now turn around and look from the sick service's side, which is the less obvious half.

What does a struggling service need in order to recover? Almost always, headroom. It needs to finish the work already queued, rebuild a cache, let a garbage collection complete, wait out a failover. What it is receiving instead is the full offered load, plus retries, and every arriving request consumes a little of the capacity that recovery requires. A service can be perfectly capable of healing in four seconds and never heal at all, because it never gets four seconds of slack. The breaker manufactures that slack by removing the load — which is why "stop calling it" helps the callee, not only the caller.

So the refusal serves two purposes at once. Do you see that they are different purposes? One protects the caller's other work; one gives the callee room to recover. A design that only did the first would still leave the dependency pinned down.

Now, how should such a thing behave? The obvious design is a switch — but a switch that only opens is useless, and a switch a human must close is too slow. So think about what the mechanism needs. It needs to notice sustained failure, which means counting failures over a window rather than reacting to one; a single timeout is normal life. It needs to stay open for long enough to be useful — an interval measured in seconds, tuned to how long recovery actually takes. And it needs a way to find out whether the dependency has recovered, without going straight back to full load, since dumping full traffic onto a barely-recovered service simply re-breaks it.

That last requirement produces the third state, and it is the clever part of the pattern. After the cool-down, the breaker goes half-open: it permits a small number of trial calls through while continuing to reject the rest. If a trial succeeds, the breaker closes and normal traffic resumes. If it fails, the breaker opens again and the timer restarts. So the recovery is probed cheaply rather than tested with the entire load.

Three states, then: closed (traffic flows, failures counted), open (calls rejected instantly, timer running), half-open (a trial trickle decides which way to go). Every implementation differs in the details — error-rate thresholds versus consecutive-failure counts, a minimum request volume before the breaker may trip, breakers scoped per dependency or per endpoint — but those three states are common to all of them.

One thing to be honest about: a breaker only helps the user if there is something sensible to serve while it is open — a cached value, a reduced result, a queued acknowledgement — so the fallback deserves as much design thought as the threshold.

A few refinements.

A breaker is not a retry policy, and it is not a substitute for one; they operate at different scales. A retry asks "should this one request be attempted again?" A breaker asks "is this dependency worth calling at all right now?" They compose well — retries handle isolated transient faults, the breaker handles the correlated ones — and the breaker is what stops a reasonable retry policy from becoming a retry storm during a broad outage.

The most common mistake is thresholds set from imagination. Trip too eagerly and you disable a working dependency over normal noise — a breaker that opens on three consecutive failures against a dependency with a one-percent error rate will flap all day. Trip too reluctantly and the caller has already exhausted its threads before the breaker notices. The numbers must come from the dependency's measured error rate and its measured recovery time, and they should be revisited when either changes.

Finally, a breaker carries risk of its own: it is a stateful judgement about another system, and when it is wrong it causes an outage the dependency is not having. Instrument it — every breaker should emit its state changes, or you will one day debug a service refusing calls to a dependency that has been healthy for an hour.

• Bulkheads, which limit the damage a single dependency can do even while the breaker is still closed.
• What a good fallback looks like — cached, degraded, or queued — since fast failure only helps if something sensible follows it.

Every term the collection defines is gathered in the glossary.