Keeping a Long-Running Managed Agent Alive Across Sandbox Recycling — Durable Checkpoints and Idempotent Resume

Idempotent resume — never run a side effect twice

Once you add checkpoints, a new problem appears: deciding where to resume.

Say "step 7 sent an external notification, and the sandbox vanished right after." The checkpoint only made it through step 6. The next run resumes at step 7 and sends the notification again. A duplicate notification — or, if it's billing, a duplicate charge.

The key to avoiding this is idempotency. Give each step a unique idempotency key, and before running a side effect, check "has this key already been committed?" If it has, skip the execution. If it hasn't, execute, then record the commit.

Order matters. The naive "execute then record" leaves a double-execution window if you crash between execute and record. The reverse, "record then execute," marks a step done that was never actually executed if you crash after the record and before the execute. Both simple options leave a hole.

So we split it into three phases.

1. claim — reserve a row by idempotency key. If it's already committed, return the stored result and stop. If a live claim from another run exists, wait for it or take it over.
2. execute — run the side effect exactly once. The claim guarantees "not yet committed" for whoever reaches here.
3. commit — write the result and the "committed" mark in the same transaction.

The crux is that the commit is atomic. If the result and the committed mark live in one transaction, you can never produce "result exists but uncommitted" or "committed but no result." A crash can only land before or after the transaction. Before, and you resume as un-executed; after, and you skip it as committed.

Working code — verified with SQLite

Let's turn the design into something that runs. The external store is SQLite. At an indie developer's scale, this is enough to prove the pattern. If several sandboxes will touch it concurrently in production, swap in Firestore or Cloud SQL as discussed below. All you need is "an atomic store that lives outside the sandbox."

First, the checkpoint store and the receiver for committed results.

import sqlite3
import json
import time
from contextlib import contextmanager
 
class Checkpoint:
    """A durable store for progress and committed results, living outside the sandbox."""
 
    def __init__(self, path: str, run_id: str):
        self.run_id = run_id
        self.conn = sqlite3.connect(path, isolation_level=None)  # manage BEGIN/COMMIT ourselves
        self.conn.execute("PRAGMA journal_mode=WAL;")
        self._init_schema()
 
    def _init_schema(self):
        self.conn.executescript(
            """
            CREATE TABLE IF NOT EXISTS progress (
                run_id     TEXT PRIMARY KEY,
                step       INTEGER NOT NULL,
                state_json TEXT NOT NULL,
                updated_at REAL NOT NULL
            );
            CREATE TABLE IF NOT EXISTS committed (
                idem_key   TEXT PRIMARY KEY,
                run_id     TEXT NOT NULL,
                result_json TEXT NOT NULL,
                committed_at REAL NOT NULL
            );
            CREATE TABLE IF NOT EXISTS claims (
                idem_key   TEXT PRIMARY KEY,
                run_id     TEXT NOT NULL,
                claimed_at REAL NOT NULL
            );
            """
        )
 
    @contextmanager
    def _tx(self):
        self.conn.execute("BEGIN IMMEDIATE;")
        try:
            yield
            self.conn.execute("COMMIT;")
        except Exception:
            self.conn.execute("ROLLBACK;")
            raise
 
    def load(self):
        """Call on resume. Returns the reached step and the resume state."""
        row = self.conn.execute(
            "SELECT step, state_json FROM progress WHERE run_id=?",
            (self.run_id,),
        ).fetchone()
        if row is None:
            return 0, {}
        return row[0], json.loads(row[1])
 
    def save_progress(self, step: int, state: dict):
        """Update only the map. The cargo is already in the committed table."""
        with self._tx():
            self.conn.execute(
                "INSERT INTO progress(run_id, step, state_json, updated_at) "
                "VALUES(?,?,?,?) "
                "ON CONFLICT(run_id) DO UPDATE SET "
                "step=excluded.step, state_json=excluded.state_json, "
                "updated_at=excluded.updated_at",
                (self.run_id, step, json.dumps(state), time.time()),
            )

Next, the three-phase idempotent execution in a single method. It takes a function with a side effect and guarantees "exactly once."

    def run_once(self, idem_key: str, side_effect):
        """claim -> execute -> commit. The side effect runs once per idem_key."""
        # --- claim phase: if already committed, return the result and stop ---
        done = self.conn.execute(
            "SELECT result_json FROM committed WHERE idem_key=?",
            (idem_key,),
        ).fetchone()
        if done is not None:
            return json.loads(done[0]), "skipped"
 
        # Place a reservation. On conflict (another run is executing), await its commit.
        try:
            with self._tx():
                self.conn.execute(
                    "INSERT INTO claims(idem_key, run_id, claimed_at) VALUES(?,?,?)",
                    (idem_key, self.run_id, time.time()),
                )
        except sqlite3.IntegrityError:
            return self._await_commit(idem_key), "awaited"
 
        # --- execute phase: only the run that reached here executes the side effect ---
        result = side_effect()
 
        # --- commit phase: write result and committed mark in one transaction ---
        with self._tx():
            self.conn.execute(
                "INSERT INTO committed(idem_key, run_id, result_json, committed_at) "
                "VALUES(?,?,?,?)",
                (idem_key, self.run_id, json.dumps(result), time.time()),
            )
            self.conn.execute("DELETE FROM claims WHERE idem_key=?", (idem_key,))
        return result, "executed"
 
    def _await_commit(self, idem_key: str, timeout: float = 30.0):
        deadline = time.time() + timeout
        while time.time() < deadline:
            row = self.conn.execute(
                "SELECT result_json FROM committed WHERE idem_key=?",
                (idem_key,),
            ).fetchone()
            if row is not None:
                return json.loads(row[0])
            time.sleep(0.5)
        raise TimeoutError(f"timed out awaiting commit: {idem_key}")

I made run_once return "executed" / "skipped" / "awaited" alongside the result. On resume you can aggregate exactly how many steps were skipped — a small piece of instrumentation to confirm, by number rather than by impression, that double execution is really being suppressed.

Wiring it into the agent loop

The Managed Agents loop becomes "read to start, write to advance." Below is the typical shape for processing a list of inputs in step order. The API surface may shift as the public preview evolves, so confirm the current method names and arguments against the official docs before you implement. What you don't want to disturb is not the call surface but the sequence: read, run idempotently, update the map.

def run_agent(inputs, ckpt: Checkpoint, call_model):
    start_step, state = ckpt.load()   # get the resume point from the store
    processed = state.get("processed", 0)
 
    for step in range(start_step, len(inputs)):
        item = inputs[step]
        idem_key = f"{ckpt.run_id}:step:{step}"
 
        # side effect (model call + committed write) exactly once
        result, status = ckpt.run_once(
            idem_key,
            lambda: call_model(item),
        )
        processed += 1 if status == "executed" else 0
 
        # update only the map; the result is already in committed
        ckpt.save_progress(step + 1, {"processed": processed})
 
    return processed

The point is that save_progress comes after run_once's commit. With this order, a sandbox that vanishes after run_once's commit but before save_progress doesn't break anything. The next run retries the same step, run_once sees the committed row and returns "skipped", and the map quietly catches up without any extra side effect. "The map may lag a little; the cargo must never duplicate." Allowing that asymmetry is what keeps resume simple.

How fine-grained should checkpoints be?

Per-step saves are safe but write more. Batched saves (every N steps) are lighter but widen the range you redo on resume. This can only be decided by measurement.

Locally I measured a synthetic 1,000-step workload (a stub assuming an average 400ms model call per step) against SQLite (WAL). I forced an interruption at step 620 and counted the steps redone after resume.

The reading is simple. The checkpoint save itself is a rounding error next to a model call. In a world of 400ms per step, one save costs 1.2ms. Trade that away for batched saves and, on interruption, you drop tens of steps' worth of model calls. The asymmetry that matters: one redone step costs roughly 330x more than one save.

So the guidance: in loops whose side effects are "expensive to redo," like model calls, make per-step saving the default. Saves become relatively heavy only in pure-compute loops where a step finishes in sub-milliseconds, and only there should you tilt toward batching. Decide granularity by the ratio "redo cost ÷ save cost," and the choice stops being agonizing.

Operational notes the docs don't cover

A few things I only learned by running it — points that live outside the documentation.

First, sweep orphaned claims. If the sandbox dies mid-execute, a claims row is left behind. The next run can't claim the same key and waits until _await_commit times out. The fix is to give claims a claimed_at and treat any claim past a threshold as "expired," reclaimable (the lease idea). The code above is minimal, so add this expiry in production.

Second, keep idempotency keys stable across runs. I first mixed a timestamp into the key. That changed the key on every resume and voided idempotency entirely. Derive keys purely from "what is being processed" — a content hash of the input or an immutable ID — and never mix in time or attempt count.

Third, doubt the single-sandbox premise early. SQLite works only while one thing touches it at a time. The moment you start parallel runs, the external store has to move to Firestore transactions or Cloud SQL row locks. The design (three phases, atomic commit) carries over unchanged, but the store choice has to track your scale.

Fourth, count the "skips" to stay observable. The number of "skipped" results run_once returns is your evidence that resume worked correctly. Emit it as a metric, and when a double-execution suspicion arises, you can first tell a design hole apart from a data problem.

Recommendations by situation

In closing

If you take away one thing, decide this: progress outside the sandbox, side effects exactly once via idempotency keys. With those two in place, whenever the environment is rebuilt, the run quietly continues from partway through.

A good next step is to wrap run_once around exactly one spot in the long-running loop you're running today. Give an idempotency key to the single side effect you'd least want run twice. Grow it from there, and the design settles in without strain.

For me, the morning I finally erased that "back to square one," my chest felt quietly lighter. Being able to let go of the operations you can let go of — safely. I hope this helps you do the same. Thank you for reading.