Durable execution for AI agents: the engine lives in code

A production agent rarely dies at the end. It dies in the middle. The tenth step of a twenty-step flow, a human approval pending for two hours, an external system down, an API session that expired in between. The cost of the disaster is not the failure itself. It is the next question: redo everything from the start, or resume from the last valid point?

The difference between redoing and resuming is what engineering calls durable execution, and it has returned to the center of the debate with the announcement of the Workflow Development Kit from Vercel: durability as a language-level concept, ordinary asynchronous code that persists its own progress and resumes after a crash or a deploy, with no queue and no state machine in YAML. The position of this piece goes beyond the announcement: for long-running AI agents, the execution engine belongs in the code, as a property of the flow, with a checkpoint where the team decides, not as a distributed product rented alongside it.

The argument does not depend on the SDK or the vendor. It rests on sources no pitch can replace: the paper that described long-lived transactions in 1987, the one that formalized durable execution in serverless at ASPLOS 2022, and a public 2018 postmortem in which 43 seconds of network partition became 24 hours and 11 minutes of degraded service. In the end, durable execution is an accounting decision: whoever pays for the repeated steps pays for the architecture that repeats them.

Long-running agents do not fail like chat questions

A chat question that fails costs a retyped prompt. An agent flow that fails costs the steps already executed: tokens billed, calls made, hours spent waiting for approval. Long-running AI agents carry that accumulated cost, and durable execution is the property that turns redoing into resuming. Without it, every failure re-bills the entire invoice.

The failure pattern of a flow that runs for hours is of a different nature. The agent crosses sequential steps, some with external effects that are hard to undo, and it stops exactly where the outside world takes part: the manager's approval, the ERP's response, the partner's webhook. While it waits, it pays. Every context token resent is billed, and waiting for a human is not idle time, it is exposure time.

The failure is never a question of whether, only of where.

Between any two steps there is a layer that decides who calls what, with which permission and which trail, and it already has dedicated treatment in the AI agent tool-traffic control layer. The execution layer is a different one, and the least discussed of the three: it decides what happens to the step that died in the middle. The confusion is old: the same word covers both layers, and the buyer leaves the room believing they bought failure recovery for AI agents when they bought a queue.

What durable execution is in an AI agent

Durable execution is the property of a flow that writes a checkpoint at every step and resumes from the last valid point after a failure or a restart. State persists outside the process's memory, and resumption executes only what is still pending. Redoing becomes the exception; resuming becomes the default behavior.

Three pieces define the mechanism. The checkpoint writes, before the next step, the result of the completed step, the flow's variables, and the exact position within the sequence. The runtime reconstructs state from that record, not from the memory of a live process. Resumption then executes the pending step, and only it.

The academic formalization is recent, and the concept is old. Mainframe batch systems in the 1960s already wrote checkpoints and restarted the job from the last valid point. The paper Durable Functions: Semantics for Stateful Serverless, published at ASPLOS 2022, gave the concept formal semantics in the serverless world: the flow writes an event log, the runtime replays that log deterministically to reconstruct state, and resumption survives a deploy, a crash, and a change in scale.

The detail that separates the definition from practice is the order between writing and acting. A checkpoint written before the external effect opens the door to a retry that applies the effect twice. A checkpoint written after loses the entire step when the process dies between the effect and the write. A window of loss and a duplication of effect: this is where durable execution stops being a ready-made feature and becomes design.

Where the agent's state lives between steps

State lives in the flow's execution log, not in a process's memory: the result of each step, the decisions made, and the current position are written as a checkpoint before the next step. Resuming from the last valid point means executing only the pending step, without re-charging model calls that were already paid for.

The question of who pays the bill is narrower than the question of who designs the architecture: what happens to calls already paid for when the flow dies? With a checkpoint, the answer is nothing. Their results are in the log, and resumption does not repeat the call. Without a checkpoint, the answer is everything: the flow does not know what already ran, and it runs it again. The difference between the two answers is the month's token bill.

State brings a second problem, the external effect. The Sagas paper, by Hector Garcia-Molina and Kenneth Salem, published at SIGMOD 1987, described the problem before agents existed: a long-lived transaction splits into steps, and when one fails, the completed ones do not undo themselves, they demand compensating actions. Forty years later, the translation is direct: the step that issued a payment needs an idempotent mechanism, because a retry applies the effect twice, and the compensating action is the expensive plan B.

The third problem is state that diverges. GitHub's October 30, 2018 postmortem is the case study: 43 seconds of lost connectivity between two data centers became 24 hours and 11 minutes of degraded service, not because the partition was severe, but because two clusters came to hold writes the other did not have. Manual reconciliation of a few seconds of writes was still running days later, and one of the busiest clusters had 954 writes in the affected window. By the time the queue resumed, roughly 200,000 webhook payloads had expired and were discarded. Poorly managed state charges interest: the original failure lasted 43 seconds, and its consequences lasted days.

Dedicated orchestration engine or execution in code: where the real cost sits

A dedicated orchestration engine delivers queue, scheduling, and centralized visibility, and charges for it an entire distributed system to operate. Durable execution in code delivers checkpoint and resumption as a property of the flow, with whatever granularity the team chooses, and charges writing discipline. The difference shows up in the first failure, in the bill, and in how the flow adapts.

The inherited word confuses: AI agent orchestration covers, in the same breath, the queue that distributes work and the logic that recovers from failure, and those are different problems. The three cost differences live in different places. The first is resumption granularity. When a flow orchestrated by a dedicated engine fails in the middle of a step, resumption happens at the granularity the engine defines: in the cases where it does not carry the step's state at the point of failure, the whole segment runs again, paid calls included. Execution in code places the checkpoint where the developer decides, and resumption redoes only what is pending.

The second is the cost of idle steps. An agent flow spends most of its life waiting: a human approval, an external system's response, a processing window. The dedicated engine is infrastructure that runs, and costs, even with every flow under its watch on hold. Durable execution runtimes designed to suspend the flow between steps spend no resources on the wait, and the infrastructure left over is what the company already operated. At scale, it becomes a line in the budget.

The third is the adoption curve. A flow written as ordinary code enters the repository, the code review, the tests, and the CI the company already runs. A flow written in the semantics of a dedicated engine carries a second distributed system to operate, with its own versioning, its own topology, and its own incidents. The 2018 postmortem shows the characteristic failure mode of that transfer: the failover mechanism acted exactly as configured, and it was the application that could not support the resulting topology. Dedicated tooling pulls the semantics inside itself; code leaves the semantics with whoever writes it.

What remains is the observability row, the most misread one in the table. A dedicated engine's dashboard shows the queue: how many executions, in which state, waiting for how long. It shows that the flow stopped. It does not show whether the current step's decision was a good one, and that is what agent evaluation covers: what to measure beyond the final answer. A visible queue is not audited behavior.

When a dedicated engine still wins

At loads of thousands of parallel flows with contractual service-level guarantees, a dedicated orchestration engine delivers what plain code alone cannot: centralized fleet visibility, scheduling proven in production, and a product with an audit trail ready for compliance. In that configuration, the cost of operating the engine pays for itself.

The opposite side has a strong version too. A platform team operating hundreds or thousands of flows in production needs to see the whole fleet in one place, respond to incidents with a known procedure, and show an auditor a recognizable control surface. A mature dedicated engine sells exactly that, and a company with that team and that load buys a product instead of building an internal platform. It is the right purchase for whoever has a fleet problem.

The argument falls for the majority of B2B agent workloads, and it falls by arithmetic. An agent flow at a typical company is counted, not a fleet: one reconciliation, one contract triage, one procurement approval. It changes every week, because the business rule, the prompt, and the connector change. The team behind it is rarely a dedicated platform. For that profile, the fleet is too small to justify the product and changes too fast for the rigidity of someone else's semantics. The practical criterion: a fleet of thousands with a platform team buys an engine; a fleet of a few with a lean team writes the checkpoint in code.

What this decision changes for buyers of B2B agents

Durable execution stops being an engineering detail and enters the contract: where the checkpoint lives, what happens to the step in the middle, who pays for the repeated calls, and how the resumption is audited. An agent without answers to these four questions is not ready for production, no matter how good the model underneath is.

The four questions, with what each answer commits:

1. Where does the checkpoint live? Who owns the state: the flow, in storage the company operates, or the product's internal structure. The answer decides who touches the granularity of resumption.
2. What happens to the step in the middle? What is the unit of resumption: the step, the segment, or the entire flow. The answer shows up in the bill of the first failure.
3. Who pays for the repeated calls? If resumption redoes model calls already paid for, the cost of failure doubles, and nobody budgeted that line.
4. How do you audit the resumption? The post-failure trail needs to show what ran again, what was compensated, and what reached the customer twice.

In the Nexforce Agents stack, the decision has an address at both ends. Nexforce Code is the developer runtime: agents and subagents defined per project, with skills, project rules, and MCP support for external tools and data, running in the terminal and the IDE and also in headless executions for automation and CI, on macOS, Windows, and Linux. Nexforce Work is the business team's workspace: a desktop where non-technical teams run agents over their own files, tools, and connectors, with multi-workspace orchestration, an approvals and permissions layer, reusable workflow templates, sandboxed execution, and scheduled runs for recurring routines.

Work's approvals are the exact point where a long-running flow pauses. Scheduled executions are the recurring side. MCP connectors are the external systems that fail in the middle. And where the checkpoint lives remains flow design: whoever writes the agent decides the granularity of resumption.

Underneath, the model layer is the Nexforce Router: hundreds of models behind a single API, with automatic failover between providers when one of them goes down, because a flow's resumption needs an available model to come back to.

The architecture decision also enters the business case: the ROI of AI agents is measured before scaling the pilot, and the cost of redoing is one of the lines almost nobody includes in the spreadsheet.

FAQ

How does an AI agent recover from a failure in the middle of a long flow?

The agent reads the last valid checkpoint and executes only the pending step. The checkpoint carries the results of completed steps and the position in the flow, so resumption does not repeat calls already paid for. For sensitive external effects, such as an issued payment, the step needs to be idempotent, because a retry applies the effect twice.

How do I validate that my agent's durable execution works before scaling?

Validation means failing on purpose: a failure is injected between two steps, resumption is checked at the correct checkpoint, no repeated call shows up on the bill, and the trail shows what ran again. GitHub's 2018 postmortem ended in a formal fault injection initiative. Durable execution not demonstrated is durable execution assumed.

Does an agent that runs in minutes also need durable execution?

It depends on the cost of redoing. A single-call flow redoes cheaply, and a checkpoint there is bureaucracy with no return. A flow with a human approval, an external system in the middle, or per-token billing pays for resumption from the first failure. The criterion is the cost of rework, not the flow's chronological duration.

Does durable execution replace the control layer and the evaluation layer?

No. Failure recovery in AI agents is one of the three layers of the operation: control decides who calls what and with which permission, evaluation decides whether the behavior is good, and durable execution decides what happens when the flow dies in the middle. An agent that resumes perfectly and decides poorly keeps deciding poorly.

Does durable execution require a dedicated engine?

It is not a requirement. The checkpoint can be written by the flow's own routine, in storage the company already operates, with whatever granularity the team chooses. A dedicated engine adds queue, scheduling, and centralized visibility, and charges an entire distributed system for the convenience. The choice is one of operating cost, not technical possibility.

References and Further Reading

Four primary sources support the argument.

• Built-in durability: Introducing the Workflow Development Kit: the occasion of the debate over where durable execution lives.
• Durable Functions: Semantics for Stateful Serverless, ASPLOS 2022: the formalization of durable execution semantics in serverless.
• Sagas, Hector Garcia-Molina and Kenneth Salem, SIGMOD 1987: long-lived transactions, steps, and compensating actions.
• October 21 post-incident analysis, GitHub, October 30, 2018: the public postmortem in which 43 seconds of partition became 24 hours and 11 minutes of degraded service and manual state reconciliation.

The decision that stays

The next conversation about agents, at the architecture table or the buying table, changes its question. Instead of "which engine do you use", the question that decides is "show me the checkpoint": where it lives, when it is written, and what happens to the step after the failure. Redoing is the expensive answer. Disguised as simplicity, it arrives with the first outage of an external system in the middle of a twenty-step flow. Durable execution written into the flow itself is the version where the agent resumes the overnight shift with nobody on call. And resumption has an address: code is the only place where the granularity of resumption belongs to whoever pays the bill.