Durable execution

How does a process survive the machine it was running on disappearing mid-step?

A program is halfway through a long piece of work. It has charged a card, it has not yet booked the seat, and the machine it is running on vanishes — a spot instance reclaimed, a kernel panic, a datacentre rack losing power. When something starts up to take over, what exactly does it have to work with?

The tempting answer is "it restarts the program". But that is precisely the thing we must not do naively, because restarting means charging the card again. So the real question hiding underneath is narrower and stranger: what is the surviving part of a running process, once the machine holding it is gone?

Let us ask what was lost. A running process holds two quite different sorts of thing. There is the code — the instructions, identical on any machine, trivially reconstructible. And there is the state — the program counter saying which line we reached, the local variables, the contents of the stack. The code was never at risk. Only the state was, because only the state lived in memory on that one machine.

So what if the state were somewhere else? Notice that we do not need to preserve memory itself. We need to be able to reconstruct it. And here is the observation the whole idea rests on: if the program is deterministic, then its state at any moment is completely determined by two things — where it started, and every result it received from the outside world along the way. Same starting point, same sequence of answers, same state. Nothing else is needed.

That reframes the problem entirely, does it not? Instead of continuously copying memory to safety, we only have to record, durably, each result as it comes back: the card charge returned this confirmation id, the timer fired at this moment. Those results are few, small, and unlike memory they are meaningful.

Now play the recovery forward. A new worker picks up the abandoned workflow. It starts the code from the top — genuinely from line one — and runs it. It reaches the card charge and is about to call the payment provider. But the engine consults the record and sees that this step already completed, with this result. So it does not make the call. It hands back the recorded answer and lets the code continue, having spent no money. It does this for every step in the record, in order, until it reaches a step with no recorded result. That is the frontier — the point the old machine actually got to — and from there the code runs for real again.

The word for that reconstruction is replay, and the record is usually called the event history. What is striking is what the workflow code itself experiences: nothing. From inside the function, one line follows another. It has no idea it just executed its first eleven steps at a million times normal speed against a log, and is only now touching the world.

Two things must hold, and they are exacting rather than incidental. The history has to be written durably before the workflow is told the result — otherwise a crash in the gap loses the fact that the card was charged, and replay charges it again. And the code has to be deterministic on replay: if it asks the current time, or generates a random number, then the second run diverges from the first and the recorded results no longer line up with the operations requesting them. Engines enforce this rather than hoping for it — they intercept clocks and random sources, feed recorded values instead, and refuse to continue on divergence.

And one honest limit. Nothing here makes an external side effect exactly-once. If the worker crashed after the payment provider took the money but before the result was durably recorded, the history has no memory of it, and replay will call again. What durable execution actually guarantees is that the workflow's own progress is never lost and never silently duplicated; making the outside world agree still requires idempotency keys at the boundary. That is a division of labour worth being clear about rather than blurring.

Three refinements are worth making.

First, "durable" is doing precise work in the name. It does not mean the process kept running; the process certainly died. It means the execution — the notion of where this particular piece of work has got to — outlived every machine that ever hosted it. Durability lives in the history store, which is a replicated database, not in any worker.

Second, replay is not a retry. A retry re-attempts one failed operation; replay re-runs the entire function from its first line and suppresses everything already done. This is why log output appears repeatedly during recovery, and why engines mark replayed logging as such.

Third, the common misconception: that this is checkpointing. Checkpointing snapshots state; durable execution records results and recomputes state from them. A memory snapshot is opaque, large, and tied to a binary layout; a history of results is small, inspectable, and survives a code deploy far more gracefully — though not arbitrarily, since changing the sequence of steps in an already-running workflow is exactly the case that breaks replay, and is what versioning APIs exist to handle.

• Why the determinism requirement forbids ordinary things like reading the clock, and what an engine substitutes instead.
• How a workflow that is already mid-flight survives a deployment that changes its own sequence of steps.
• Designing idempotent boundaries, so a repeated external call is harmless.

Every term the collection defines is gathered in the glossary.