Alert fatigue

Why does adding more alerts to catch more problems make outages last longer?

A team has an outage. In the review, someone says the sensible thing: nobody was watching the thing that broke. So they add an alert for it. Repeat that after every incident for two years, and the team is watching everything — and their outages are lasting longer than they used to.

That ought to be impossible. Each alert, taken alone, was added for a good reason and detects a real condition. So where does the harm come from? Not from any one alert. It comes from what the whole set does to the person receiving it, and that is a different object of study from the alerts themselves.

Start with a question about arithmetic rather than psychology. Suppose you build a detector for a condition that occurs, genuinely, about once a month. It is a good detector: when the condition is present it fires, and when it is absent it stays quiet ninety-nine times out of a hundred. It evaluates every minute. How often does it page you falsely?

Roughly forty-three thousand evaluations a month, one percent of which are wrong — so hundreds of false pages surrounding one true one. The detector's accuracy is excellent. Its precision — the share of its pages that mean something — is close to nothing. That is not a flaw in the detector; it is what happens when a rare event meets an imperfect test. And notice this does not require the detector to be bad. Rarity alone does the damage.

Now put a person on the other end. What does someone do when the great majority of interruptions turn out to be nothing? They do not keep responding at full attention — nobody can. They begin to defer, to batch, to glance and dismiss, to assume the new page is the same as the last forty. Eventually they mute a channel, or route a class of alerts to a folder they will read later. Each of those is a rational local response to a low-precision signal, and together they are exactly what we call alert fatigue.

This has been measured most carefully in hospitals rather than in software. The Joint Commission's 2013 Sentinel Event Alert on alarm safety reported that a large majority of clinical alarm signals — their figure was between 85 and 99 percent — do not require any clinical intervention, and it linked deaths to staff having disabled or ignored alarms. Software has no comparable body of controlled evidence; what we have there is observational and experiential, which is worth admitting plainly. But the mechanism is not medical. It is a property of any low-precision signal delivered to a finite human.

Now the part that answers the question. Why would outages get longer, rather than merely more annoying? Two effects compound during the incident itself.

The first is the alert storm. One real fault rarely trips one alert. A saturated database trips the database alert, and the queue depth alert, and the latency alerts on nine services downstream, and the error-rate alerts under those. The responder does not receive a diagnosis; they receive a hundred symptoms arriving in an order set by evaluation intervals rather than by causality. The single page that would have named the cause is somewhere in that pile, indistinguishable from the ninety-nine it caused.

The second is the trained delay. Because most previous pages were nothing, the responder's first move is now to check whether this one is real before acting on it. That is learned behaviour and it is well-founded, but it inserts minutes of verification in front of every response — including the responses that mattered. Detection time goes up even when detection itself works perfectly.

So the causal chain is not mysterious: more alerts, each with weak precision, produce a stream in which informative pages are outnumbered; humans adapt by discounting the stream; and during a real failure that discounting plus correlated firing lengthens the time to a correct diagnosis. The alerts did their job. The system around them did not.

What is the remedy, then? Notice the question the reasoning has pushed us toward. It is not "what else should we watch?" but "what should page a human?" — and those come apart. Signals can be recorded without paging; dependencies can suppress downstream alerts when an upstream one is already firing; and the discipline of alerting on user-visible symptoms rather than on internal causes — checkout failing, not one node's CPU — exists precisely because symptom alerts are far fewer and far more often real.

Three refinements are worth making.

First, the common misreading: that alert fatigue is a discipline problem, cured by telling responders to take every page seriously. It cannot be. Attention is finite and the discounting is a correct inference from the observed base rate. Asking people to ignore that inference is asking them to reason badly on purpose, and it does not survive contact with a third night shift.

Second, precision and recall trade against each other, and the trade is not free. Tightening thresholds to reduce false pages will let some real events through undetected. There is no setting that removes both errors, so a team is always choosing where to sit — and the point of the argument above is only that the reflexive choice, maximum sensitivity, is usually not the one that minimises total harm.

Third, none of this argues for less monitoring. Collect everything you can afford. The scarce resource is not storage; it is the interruption of a human being, and that budget should be spent on signals that are usually true and always actionable.

• Symptom-based alerting, and why paging on user-visible failure produces so many fewer alerts than paging on causes.
• Alert dependency suppression: letting an upstream alert silence the ones it provokes.
• Base rates and positive predictive value, worked through with a screening test rather than a monitor.

Every term the collection defines is gathered in the glossary.