Your Managed Agents Bill Has a Second Axis: Drawing a Budget Boundary Around Sandbox Runtime

Fold a stalled sandbox without waiting for the cap

A wall-clock cap alone still wastes time. A run that jams after 30 seconds against a 180-second cap will sit out the remaining 150 seconds. Idle teardown closes that gap.

Each time the agent emits some kind of "progress," record the last-update timestamp; if it does not move for a set interval, treat it as hung and fold it. Progress can be a completed step, a tool call, or an output chunk — whatever your agent can observe.

async def _poll_until_done(client, run_id, budget):
    last_progress = time.monotonic()
    last_step = -1
    while True:
        run = await client.agents.poll_run(run_id)
        if run.completed_steps != last_step:
            last_step = run.completed_steps      # progress happened
            last_progress = time.monotonic()
        if run.status == "succeeded":
            return run.output
        if run.status in ("failed", "cancelled"):
            raise RuntimeBudgetError(f"run {run_id} ended as {run.status}")
        if time.monotonic() - last_progress > budget.idle_seconds:
            await client.agents.cancel_run(run_id)
            raise RuntimeBudgetError(
                f"run {run_id} torn down after {budget.idle_seconds}s without progress"
            )
        await asyncio.sleep(1.5)

In practice, set the idle ceiling well below the wall-clock cap. I put the wall-clock cap at roughly 3x a stage's measured median runtime, and the idle ceiling around that same median. Healthy runs always emit progress before the idle ceiling, so they are never folded; only genuinely stuck runs get cut early.

A concurrency ceiling stops runtime from multiplying

What makes runtime billing dangerous is that parallelism multiplies the seconds. Since each run gets its own sandbox, five concurrent runs mean five sandboxes metering seconds at once.

At an individual's budget scale, leaving this unbounded is risky. Use an asyncio.Semaphore to cap how many sandboxes can be alive at once, and queue the overflow.

async def run_batch(client, *, model, jobs, budget, ledger):
    sem = asyncio.Semaphore(budget.max_concurrent)
 
    async def _one(job):
        async with sem:                       # wait once the ceiling is hit
            try:
                output, secs = await run_with_wall_clock(
                    client, model=model, instruction=job["instruction"], budget=budget
                )
                ledger.record(job["stage"], secs, status="ok")
                return output
            except RuntimeBudgetError as e:
                ledger.record(job["stage"], budget.max_wall_seconds, status="killed")
                return {"error": str(e)}
 
    return await asyncio.gather(*[_one(j) for j in jobs])

Throughput drops a little. But being able to say "a nightly batch can never accrue more than N sandboxes' worth of seconds at once" is worth more, for an indie setup, than raw speed. The ceiling is not a performance number — it is the number that lets you sleep.

A ledger that attributes runtime to each stage

Everything above is about stopping runaways. The last piece is a ledger that lets you say which stage is eating time without waiting for next month's invoice.

On each run, append the stage name, runtime seconds, and outcome as one line of JSON. You can aggregate however you like afterward.

import json
from collections import defaultdict
from pathlib import Path
 
class RuntimeLedger:
    def __init__(self, path="runtime_ledger.jsonl"):
        self.path = Path(path)
 
    def record(self, stage, seconds, status):
        line = {"ts": time.time(), "stage": stage,
                "seconds": round(seconds, 2), "status": status}
        with self.path.open("a") as f:
            f.write(json.dumps(line, ensure_ascii=False) + "\n")
 
    def summary(self):
        total = defaultdict(float)
        killed = defaultdict(int)
        for line in self.path.read_text().splitlines():
            r = json.loads(line)
            total[r["stage"]] += r["seconds"]
            if r["status"] == "killed":
                killed[r["stage"]] += 1
        return {s: {"runtime_min": round(total[s] / 60, 1),
                    "killed": killed[s]} for s in total}

Once I started dumping summary() into a morning log, the view changed. Facts like "only the image-organizing stage has runaway runtime" or "one stage produces a killed run every single day" surfaced from my own ledger before they ever reached an invoice. A stage that produces killed runs steadily is not a signal to loosen the cap — it is a signal to question that stage's design.

Operational notes the docs do not cover

Preview docs teach you how the feature works, but not how to defend the runtime axis. A few things that only showed up once I ran it for real:

First, even after cancel_run, there is a small delay before teardown takes effect. Record runtime seconds as the client-side measured wall clock in your ledger, so you can later reconcile it against what billing actually metered.

Second, if idle detection leans only on output chunks, a model that thinks for a long stretch can be folded by mistake. Pair it with coarser progress signals — completed steps or tool calls — for stability.

Third, apply the concurrency ceiling at the account level, not per site or per project. Split the ceiling per site and the moment several sites run on the same morning, the combined sandbox count climbs and the multiplication returns. I made exactly that mistake once and allowed twice the intended number of concurrent sandboxes.

Where to start, by situation

You do not need all of this at once. Add boundaries in the order that pays off, matched to how your automation actually behaves.

Managed Agents genuinely takes the heavy part of the agent loop off your hands. In exchange, it adds one axis — runtime — that is hard to watch. Draw a boundary around that axis from the start, and you keep the convenience without going pale in front of an invoice.

Start with a single wall-clock cap that includes cancel_run, and build from there. I hope it brings a quiet bit of reassurance to anyone running the same kind of automation.