Unknown unknowns

Why can a system with every dashboard green still be failing in a way nobody thought to chart?

Every dashboard is green. Latency is nominal, error rate is flat, CPU is comfortable, the queue is short. And customers are writing in to say the product is broken.

The instinct is to distrust the customers, or to hunt for the one missing chart. But pause on something more basic first: what is a dashboard, actually? It is a set of questions somebody thought to ask, frozen into charts. Green does not mean the system is healthy. Green means the failures anyone anticipated are not occurring. Those are different claims, and the gap between them is where this whole subject lives.

Consider how a chart comes to exist. Someone imagines a way the system could go wrong — it could get slow, throw errors, run out of memory — and adds an instrument to detect that. Every dashboard is therefore a record of past imagination: a museum of the failures we have already thought of, many of them there because they already happened once.

So ask the uncomfortable question: what proportion of the ways this system can fail did anyone enumerate in advance? For a system of any real complexity, nobody knows — and that is not a rhetorical flourish, it is the actual epistemic position. You cannot count a set you cannot enumerate.

This gives a useful three-way split. Known knowns: things you measure and understand. Known unknowns: things you know you cannot see — the payment provider's internals are opaque to you, and you can at least plan around that awareness. And unknown unknowns: failure modes nobody has conceived of, which therefore have no instrument, no threshold, and no alert. The third is not a gap in a dashboard. It is a gap in the set of dashboards, and no amount of staring at existing charts reveals it.

Where do unknown unknowns come from? Two sources are worth separating. Some come from interactions: component A is fine, component B is fine, and their combination under one load pattern is not, because nobody modelled the pair. Others come from drift: the system was understood two years ago, and has since been changed by fifty people, none of whom held the whole model. Both are properties of complexity rather than carelessness, which is why "be more thorough next time" is not a remedy.

Here is the step that changes practice. If you cannot pre-enumerate the failures, you cannot pre-enumerate the metrics either — so what could you possibly do instead? The answer that observability practice converged on is to change what you collect. Instead of pre-aggregating into a fixed set of counters, keep rich, high-dimensional records of individual events — a wide structured event or span per unit of work, carrying every attribute you have: which user, which build, which region, which feature flags, which downstream calls. Then, at investigation time, slice by dimensions nobody chose in advance.

Do you see why that helps? A counter of "error rate by service" can only ever answer questions about error rate by service. But a corpus of events carrying forty attributes each can answer "which combination of attributes is unusual among the slow requests?" — a question formulated after the trouble appeared. The shift is from asking predetermined questions quickly to being able to ask arbitrary questions at all. That property is often called explorability or high cardinality support, and it is the practical answer to the unknown unknown.

It is not a complete answer, and the limits matter. You can only slice by attributes you recorded, so instrumentation coverage still bounds you — unknown unknowns are pushed back, not abolished. Aggregation remains an enemy: an average hides a bimodal distribution, so a system where five percent of requests fail utterly can present as a healthy mean. And the customer complaint is itself a signal channel, often the highest-fidelity one available for genuinely novel failure — treating support tickets as telemetry rather than as noise is a real technique, not a consolation prize.

There is also a class of failure no telemetry catches: the system doing exactly what it was told, correctly and quickly, where what it was told is wrong. A pricing rule with an inverted comparison produces clean, fast, successful responses. No instrument on the machine's own behaviour can flag that, because the machine is not misbehaving.

Three clarifications.

The first corrects the most tempting misreading: that the fix is more dashboards. Adding charts extends the museum. It converts specific unknown unknowns into known knowns one at a time, which is genuinely worth doing after each incident, but it cannot in principle close a set you cannot enumerate. Coverage of the imagined does not imply coverage of the possible.

The second is about what "green" is entitled to mean. A green dashboard supports one honest inference — the specific conditions we defined as unhealthy are not currently met — and no more. Read that way it remains useful. Read as "the system is fine" it becomes actively harmful, because it converts absence of evidence into evidence of absence and gives a team confidence exactly when it should be curious.

The third is the distinction between monitoring and observability. Monitoring watches known signals against known thresholds and is excellent at what it covers. Observability is the property of being able to answer new questions about a system's internal state from its outputs, without shipping new code to do it. You want both, and confusing them leads teams to buy more of the first when they needed the second.

• Wide structured events versus pre-aggregated metrics, and what each costs to store.
• Chaos engineering as deliberate manufacture of unknown unknowns before they happen to you.
• Support tickets as a telemetry channel, and how to route them like an alert.

Every term the collection defines is gathered in the glossary.