Cold starts

Why is the first request to an idle service slow in a way the next thousand are not?

You call a function that has not been used for a while and it takes a second or two. You call it again and it answers in fifteen milliseconds. The request did not change, the code is identical, the network is the same. The only difference is that something ran recently.

That is a peculiar kind of slowness. It is not load-dependent — the system was completely idle, which usually means fast — and it is not random. It is a cost attaching to the first of a run and to no other member of it, which says a one-time expense is being paid and then amortised. Working out which expenses those are explains both why the effect exists and why most of the advice about it aims at the wrong half.

Start with what must exist before your code can handle a request at all. A machine with capacity, which must be selected. An isolation boundary, which must be created. A language runtime, which must be started. Your code, which must be fetched from storage. Only then does your program begin.

Every one of those steps produces something reusable: the sandbox can serve many requests, the runtime once booted stays booted. So the platform keeps the assembly alive after the request finishes and routes the next one to it — and, since idle capacity costs money, reclaims it after a period of inactivity. That is the entire mechanism. There is no penalty for being idle; there is a setup cost that idleness caused to be discarded.

Now the part that people miss. Divide the setup into two halves and ask who controls each.

The platform's half is placement, sandbox creation, runtime boot, and code fetch, and its magnitude depends on the isolation technology. A V8 isolate — Cloudflare Workers' model — shares one process and one already-running runtime between many tenants, so starting one takes milliseconds. A microVM such as Firecracker, the basis of AWS Lambda, boots a stripped kernel in the low hundreds of milliseconds. A large container image that must be pulled is slower again. This half is the platform's engineering problem, and the half that has improved most.

Your half is initialisation: module imports, framework bootstrap, configuration parsing, SDK client construction, opening a connection, warming a cache, loading a model. Notice how little of this is proportional to your business logic — importing a large SDK to make one API call costs the same as importing it to make a thousand.

Which half dominates? On most real services, the second — and by a wide margin on runtimes with expensive startup semantics, where a heavyweight framework doing classpath scanning and reflective wiring can spend seconds before your handler's first line. That is why the effective remedies are unglamorous: import less, import lazily, defer anything the first request does not need, move work to build time. The platform half you mostly cannot touch; the half you can is usually the larger one.

So what do the platform features do? Provisioned concurrency keeps instances initialised so requests never meet a cold one — it does not make the start faster, it pays for it in advance, converting a latency problem into a bill. Snapshot approaches such as Lambda's SnapStart take a memory image after initialisation and restore from it, which is why they help most where initialisation is heaviest; the catch is that the image carries whatever was in memory when taken, so anything unique per instance — a random seed, a cached credential — must be regenerated on restore.

One last observation, because it reframes the problem. A cold start is not exceptional but the normal cost of elasticity: any system that scales to zero reconstructs itself on demand, and any system that scales up cold-starts every new instance precisely when it is busiest. The cold path is the traffic-surge path, and a service whose initialisation takes four seconds cannot absorb a spike however fast the platform allocates sandboxes.

Three refinements, and one measurement warning.

First, "cold start" names two things that get conflated. Platform setup time and your initialisation time are measured separately on most platforms — AWS reports an init duration distinct from invocation duration — and they respond to entirely different interventions. Any discussion that does not say which half it means is not actionable.

Second, keep-alive pings are a folk remedy with real limits. Keeping one instance warm does nothing for the fortieth instance a burst creates, and platforms recycle instances regardless of traffic. It reduces the symptom's frequency in low-traffic services and does not address the case that hurts most.

Third, the numbers move. Published cold-start figures are snapshots of a moving target. Trust the ordering — shared-runtime isolates faster than microVMs faster than large container images — and measure your own service rather than importing someone else's milliseconds.

The warning: cold starts live in the tail, so a mean hides them almost perfectly. A service where one request in two hundred takes two seconds looks healthy on averages and is visibly broken to the users who hit it. Watch the high percentiles, during scaling events rather than at rest.

• How snapshot restore interacts with anything that must be unique per instance, such as seeds and cached credentials.
• Whether provisioned concurrency is cheaper than the effort of shrinking initialisation.

Every term the collection defines is gathered in the glossary.