Durable Execution: Resuming Long-Running Agents

A short agent call finishes before anything can go wrong. A long one does not. An agent that runs for twenty minutes across forty tool calls will, sooner or later, get hit by a deploy, an OOM kill, or a request timeout partway through. When that happens, you do not want it to start over from the goal and redo every tool call it already ran.

This article is about surviving that. The core idea is simple: persist enough state that the agent can pick up exactly where it stopped. The hard part is doing it without double-charging a credit card or sending an email twice on resume.

The problem: agents outlive the process that runs them

The agent loop from what an agent actually is lives entirely in memory. The messages list, the current step count, the pending tool call: all of it exists only inside one running process. Kill the process and it is gone.

For a five-second request that is fine. For a research agent, a data migration, or a multi-step booking flow that runs for minutes to hours, it is a real problem. Over that window, the odds of an interruption stop being an edge case:

A deploy rolls the pod the agent is running on.
The container hits its memory limit and gets killed.
An HTTP request that started the agent times out at the gateway.
The machine simply crashes.

Restarting from zero is expensive in two ways. You pay for every already-completed model and tool call again, and you re-run side effects that already happened. If step 12 charged a card, replaying from step 1 charges it again. Durable execution avoids both.

Persisting state so the agent can resume

The fix is to treat the agent’s in-memory state as something you save, not something you assume will stay alive. After each turn of the loop, write the state to durable storage. On restart, load it back and continue.

What is “the state”? For a basic agent, less than you might think:

The message list so far, which is the full record of the goal, every model response, and every tool result. This is the conversation the model needs to keep going, discussed in state and memory.
The current step number, so you can enforce the step limit across restarts, not just within one process.
A run ID so you can find this specific run’s checkpoint later.

That is a checkpoint. Storage can be anything durable and transactional: a Postgres row, a Redis key with persistence on, an S3 object. A row keyed by run ID works well because the whole checkpoint is one JSON blob you overwrite each turn.

flowchart LR
    START([Goal]) --> LOAD[Load checkpoint<br/>or start fresh]
    LOAD --> PLAN[Model plans<br/>next step]
    PLAN --> ACT[Run tool]
    ACT --> SAVE[Save checkpoint<br/>to storage]
    SAVE --> PLAN
    SAVE --> DONE([Done])

The one rule that makes this work: save after every turn, before the next model call. If you crash between turns, the worst you lose is the single turn in flight, not the whole run. Save too rarely and a crash costs you more work than it needs to.

Why checkpoint the whole message list every turn instead of a diff?

A diff sounds cheaper: only write the new messages since last time. In practice the full-blob write is simpler and almost always fast enough.

The message list is text. Even a long agent run is kilobytes to low megabytes, well within a single row or object write. Overwriting the whole blob means resume is a single read with no reconstruction step, and there is no risk of a half-applied diff leaving you with a corrupt state. Reach for diffs or append-only logs only when your runs get large enough that the full write shows up in your latency budget. Start with the whole blob.

Durable execution engines: the loop as a replayable workflow

Rolling your own checkpointing works, and for many agents it is enough. But there is a whole class of infrastructure built for exactly this problem: durable execution engines. Temporal is the best-known; others in the same family are Restate, Inngest, and DBOS. If you have used a background job queue, this is the next step up.

Here is the mental model. Instead of writing a loop that holds state in memory, you write your agent as a workflow function. The engine runs that function, and every time it calls out to the outside world (a model call, a tool call), that call goes through the engine as an activity. The engine records the input and the result of every activity to a durable log.

The payoff is automatic recovery. If the process dies, the engine starts the function again somewhere else and replays it from the top. When replay reaches an activity that already completed, the engine does not run it again; it hands back the recorded result. Your code re-executes, but the expensive and side-effecting calls do not. Execution fast-forwards to where it stopped, then continues live.

sequenceDiagram
    participant W as Workflow (agent loop)
    participant E as Durable engine
    participant M as Model + tools
    W->>E: run activity: model call
    E->>M: execute
    M-->>E: result
    E-->>W: result (also logged)
    Note over W,E: process crashes, restarts
    W->>E: run activity: model call
    E-->>W: replay logged result, no re-run

To your code, the crash is invisible. You write a normal-looking loop; the engine makes it durable. You give up some freedom (activities must be deterministic on replay, and there are rules about time and randomness) in exchange for not writing checkpoint plumbing yourself.

Idempotency: replay is only safe if tools do not double-apply

Whether you roll your own checkpoints or use an engine, the same trap waits for you. Replay is only safe if re-running a tool call does not repeat its side effect. This is the idempotency work from retries and failures, and durable execution makes it non-optional.

Think about the crash window. The agent ran a charge_card tool. The charge succeeded at the payment provider. But the process died before the result was written to the checkpoint. On resume, the loop re-runs charge_card. Now the card is charged twice.

A durable engine narrows this window by logging results, but it does not close it on its own. The engine can crash after the external charge and before writing the result to its log, and it will replay the activity. The fix lives in the tool, not the framework: make the operation idempotent.

The standard technique is an idempotency key. Derive a stable key for the operation (for example, the run ID plus the step number), pass it to the downstream system, and let that system dedupe. Most payment and email providers support this directly.

def charge_card(run_id, step, amount, card_token):
    # Same run + step always produces the same key, so a replay
    # after a crash is deduped by the provider, not charged twice.
    idempotency_key = f"{run_id}:{step}:charge"
    return payments.charge(
        amount=amount,
        card_token=card_token,
        idempotency_key=idempotency_key,
    )

If the downstream system cannot dedupe, do it yourself: write a “step N done” marker in the same transaction as the checkpoint, and skip the tool if the marker is already there.

A checkpoint and resume sketch around the loop

Here is the earlier agent loop with checkpointing added. The changes are small: load at the start, save each turn, and skip work that is already recorded.

def run_agent(run_id, goal, tools, max_steps=50):
    state = store.load(run_id)              # resume if a checkpoint exists
    if state is None:
        state = {"messages": [{"role": "user", "content": goal}], "step": 0}

    while state["step"] < max_steps:
        response = model.call(state["messages"], tools=tools)
        state["messages"].append(response.message)

        if not response.tool_calls:
            store.save(run_id, {**state, "done": True})
            return response.text

        for call in response.tool_calls:
            # run_id + step feed the tool's idempotency key
            result = tools[call.name](run_id, state["step"], **call.arguments)
            state["messages"].append({
                "role": "tool", "tool_call_id": call.id, "content": str(result),
            })

        state["step"] += 1
        store.save(run_id, state)           # durable point: crash here loses nothing

    raise StepLimitExceeded(run_id)

Two things carry the whole design. store.load(run_id) at the top means a second call with the same run ID resumes instead of restarting. store.save(run_id, state) at the bottom of each turn is the durable point: cross it, and that turn is safe. The tools take run_id and step so they can build idempotency keys and stay safe across replays.

Common beginner mistakes

Saving too late: checkpointing only at the end means a crash loses the entire run. Save every turn.
Non-idempotent tools: if charge_card or send_email can double-apply, resume silently duplicates side effects. Fix the tool, not the loop.
Keeping the step limit in memory: track step in the checkpoint, or a resumed run can blow past your cap.
Storing state in a non-durable place: an in-memory dict or a Redis instance without persistence disappears on the exact crash you are guarding against.
Reaching for Temporal on day one: a Postgres row and a save call solve most agents. Adopt an engine when you have many concurrent long runs or need visibility into them.

Questions you will face in production

“Do I really need Temporal, or is a database row enough?” Start with the row. Checkpoint the message list and step to Postgres each turn and you get resume with a few lines of code and no new infrastructure. Move to a durable execution engine when you are running many long agents at once, need retries and timeouts handled for you, or want a UI to inspect stuck runs. It is a scaling decision, not a starting point.

“How do I not double-charge on resume?” Make every side-effecting tool idempotent with a key derived from the run ID and step. The provider dedupes the repeat. If the provider cannot, record a per-step “done” marker alongside the checkpoint and skip completed steps on replay. This is the single most important habit for durable agents.

“What about the model call itself: is replaying it wasteful?” A re-run model call costs tokens but has no external side effect, so it is safe to repeat. A durable engine avoids even that cost by replaying the logged response. If you roll your own, the model call sits after your checkpoint load, so a resumed run only re-runs the turns after the last saved one, not all of them.

What to remember

A long-running agent will be interrupted; design for resume, not for a clean run.
Checkpoint the message list plus the step number to durable storage after every turn.
Resume is load(run_id) at the top of the loop instead of starting fresh.
Durable execution engines turn the loop into a replayable workflow and skip already-completed calls on restart.
Replay is only safe if tools are idempotent; use an idempotency key on every side effect.
Start with a database row; adopt an engine when scale or visibility demands it.

What to study next

Durable execution keeps a long agent alive across failures, but a resumed run that quietly loops or burns tokens is still a problem you cannot see. Next, read observability and cost control for agents to learn how to trace what a long run actually did and cap what it spends.