Metric cardinality explosion
A Socratic walk-through of metric cardinality explosion — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why can adding one harmless-looking label to a metric cost more than the service it measures?
An engineer adds one label to an existing counter. The counter already records HTTP requests broken down by endpoint, method, and status code. The new label is customer_id, because it would be useful to see which customers are getting errors. It is one word in one line of code, it ships on a Tuesday, and by Thursday the metrics backend is falling over and the monitoring bill has passed the bill for the service being monitored.
How can a label do that? Nothing about the traffic changed. The same requests arrive at the same rate. The suspicion worth chasing is that a metrics system does not store a metric at all — it stores something else, and the label is not an annotation on it but a multiplier of it.
Reasoning it through
REASONING #Ask what a dimensional metrics system actually keeps on disk. Not "the request counter". It keeps one independent time series for every distinct combination of label values, each with its own name, its own index entry, and its own sequence of timestamped numbers stretching back through the retention window.
So count them. Suppose there are 200 endpoints, 5 methods, and 12 status codes in play. That is 200 times 5 times 12 — twelve thousand series. Each one is a small object; twelve thousand of them is a perfectly ordinary workload.
Now add customer_id with fifty thousand distinct values. What is the new count?
The multiplication is the whole story: 200 x 5 x 12 x 50,000 is six hundred million series. Not six hundred million data points — six hundred million series, each of which then collects a data point on every scrape interval for as long as you retain it. The engineer added one label. The system's working set grew by a factor of fifty thousand.
Notice the shape of that growth, because the popular name for it is slightly wrong. Within one metric, cardinality is the product of each label's distinct values, so adding a label multiplies rather than adds. It is exponential in the number of labels only when each new label brings a comparable number of values; the honest description is multiplicative, and that is quite bad enough. One label with fifty thousand values does more damage than ten labels with three values each.
Then ask where the cost actually lands, because this is not simply about disk. Most of the pain is in memory and indexing. A system like Prometheus holds an inverted index from label pairs to series and keeps a live in-memory chunk per active series; series count, not sample count, governs its footprint. Queries suffer too: an aggregation like sum by (endpoint) must touch every matching series before collapsing them, so a query that was cheap over twelve thousand series is doing fifty thousand times the work for an identical answer. And in a hosted service, billing is very often per active series — which is why the invoice can outgrow the compute it observes.
Now the reflective question. Which labels are dangerous? Not the ones with many values today — the ones whose value count grows with something you do not control. Customer ID grows with your success. User ID, session ID, request ID, and a full URL path with identifiers embedded grow with traffic itself, giving effectively unbounded cardinality: a new series for nearly every request, receiving one sample and never written to again. Raw exception strings do the same thing quietly. The test to apply before adding a label is not "is this useful?" but "what bounds the number of distinct values this can ever take, and who controls that bound?"
And if nothing bounds it — is the information simply lost? No, and this is the resolution worth carrying away. High-cardinality identity belongs in a different storage model. Traces and wide structured events store one record per request with as many attributes as you like, because they never pre-aggregate and so never form a cross-product. Metrics are cheap precisely because they aggregate: they throw identity away in exchange for constant cost per series. Asking a metric to carry per-customer identity is asking it to abandon the one property that makes it cheap.
One honest note on the numbers. Practical limits vary a great deal by system, so treat any specific threshold as something to measure in your own environment rather than as a universal figure. The multiplicative structure, however, is common to all of them.
The analogy
THE ANALOGY #Think of a filing cabinet where every distinct combination of properties gets its own physical folder, opened permanently the first time it appears. Sorting sales by region and by product line gives a manageable wall of folders. Add "and by individual customer", and you have committed to one folder per region per product per customer — and to opening a new empty folder every time a customer appears for the first time. The information you wanted was modest. The filing scheme you chose to hold it was not.
folders sit idle at almost no cost once created, whereas a time series keeps consuming memory, index space, and money for the whole retention window even after it stops receiving any new data — so the damage from a cardinality spike persists long after the traffic that caused it has gone.
Clarifying the model
THE MODEL #Three refinements.
The first corrects the natural but wrong mental model that a label is metadata attached to one metric. It is a coordinate. The metric name plus its full label set is the series identity, so labels do not describe series, they enumerate them.
The second concerns where cardinality is decided. It is not decided by the dashboard that displays the metric, or by the query that reads it — it is decided at write time, by the instrumentation. You cannot get the cost back by not querying it. This is why cardinality problems are usually shipped in a code change and discovered by an invoice.
The third is about the remedy. The reflex is to shorten retention or drop the metric wholesale; the better move is usually to bound the label. Replace a raw path with a route template so /orders/8817 becomes /orders/{id}. Bucket a continuous value. Keep an allowlist of label values and map everything else to other. And where per-entity detail genuinely matters, move that question to traces or events, where identity is the storage model's native unit rather than its multiplier.
A picture of it
THE PICTURE #How to readThe root is the single metric. Each branch is one label, and the series count is the product of the branch sizes, not their sum — so read across multiplying as you go. The first three branches are bounded by something the team controls. The fourth is not, and that is the whole difference: it turns a twelve-thousand-series metric into a six-hundred-million-series one. The last branch names where that information should live instead.
What became clearer
WHAT CLEARED #A metric's cost is not governed by how often it is incremented but by how many distinct label combinations exist, and labels combine by multiplication. So the question to ask of any new label is not whether the breakdown is useful, but what bounds its distinct values — because an unbounded label converts a cheap aggregate into a per-request record while still pretending to be a counter.
Where to go next
ONWARD #- Metrics, logs, and traces as three storage models with different cardinality economics.
- Exemplars: attaching a sample trace ID to a metric bucket without exploding the series count.
- How relabelling and metric filtering at the scrape or agent layer contain a spike already shipped.
Key terms
TERMS #| Term | What it means |
|---|---|
| Cardinality | the number of distinct time series a metric produces, equal to the product of its labels' distinct value counts. |
| Unbounded label | a label whose distinct values grow with traffic or with entities you do not control, such as a user, session, or request identifier. |
| Route template | a normalised path such as /orders/{id} used in place of raw URLs to keep endpoint cardinality bounded. |
Every term the collection defines is gathered in the glossary.