◉GEMINI LABJP
●CLI 0.71 — Gemini CLI v0.71.0 is out as a preview. The stable release is still v0.70.0●TOOL RETR — Tool retrieval (Preview) lets the model load tool definitions only when it needs them●2.5 ROUTE — From 12/1, gemini-2.5-pro routes to 3.8 Flash and 2.5-flash to 3.5 Flash-Lite, billed at the new rates (9 days left)●OAUTH — Users report a stored MCP OAuth token being sent to a repointed server URL without asking●PICK — Choosing a daily Gemini 4 model gets easier when you weigh reasoning, long context and price●AG NEON — Antigravity App v2.34.0 adds Gemini 4 Neon for Gemini API keys●CLI 0.71 — Gemini CLI v0.71.0 is out as a preview. The stable release is still v0.70.0●TOOL RETR — Tool retrieval (Preview) lets the model load tool definitions only when it needs them●2.5 ROUTE — From 12/1, gemini-2.5-pro routes to 3.8 Flash and 2.5-flash to 3.5 Flash-Lite, billed at the new rates (9 days left)●OAUTH — Users report a stored MCP OAuth token being sent to a repointed server URL without asking●PICK — Choosing a daily Gemini 4 model gets easier when you weigh reasoning, long context and price●AG NEON — Antigravity App v2.34.0 adds Gemini 4 Neon for Gemini API keys
Articles/API / SDK
◈ API / SDK/2026-06-28Advanced

The Morning a Managed Agent Stalled and Left No Trace — Building a Run-Observability Layer Outside the Sandbox

With Gemini Managed Agents, the sandbox lives on Google's side, so when a run stalls there is nothing left in your own logging stack. This is a working TypeScript design for an outside observability layer that taps stream events into a ledger, detects silent stalls, and folds runs into readable postmortems.

Gemini API256Managed Agents8Observability4Reliability4Cloudflare Workers6TypeScript8

✦ Premium Article

One morning, a process I run autonomously overnight had produced no output. The schedule log showed a "started" entry but no "completed." I only noticed more than twenty minutes later, when the next scheduled check ran — and by then, nothing remained anywhere to explain what had happened.

When I tried to track down the cause, I hit a wall. The work itself runs inside Gemini's Managed Agents — that is, inside Google's isolated sandbox. On my side I only had the one line that launched the agent and the receiver that was supposed to take the result. My usual logging stack records only what happens inside my own process, so about the moment it went quiet inside the sandbox, it had nothing to say.

This article is the design I landed on for that "stalls quietly" failure: building an observability layer outside the sandbox so a run can be traced after the fact. I write only the shape that actually held up, from the position of an indie developer running nightly batches unattended.

What "stalling quietly" actually looks like

A failure that throws is the easier kind. An exception flies, a stack trace stays behind, an alert fires. The nasty one is the failure where no exception appears and progress simply stops.

Managed Agents run planning, tool calls, and code execution inside the sandbox. Somewhere in there, an external API times out silently, the agent falls into a loop redoing the same move, or it waits forever on a tool result that never returns. When that happens, the operation transitions to neither "failed" nor "succeeded" — progress just stops arriving. From the receiver's side, all you can see is "not done yet."

In my setup, this silent stall going unnoticed until morning was the real pain. If it crashes, I can retry; but something that stops and never returns sits there until a human thinks "that's odd." The first requirement I put on the observability layer was not flashy visualization — it was simply this: don't let a silent stall stay silent.

Why your usual logging stack can't follow it

LLM observability tools — distributed tracing, error collection, and the like — are all built on the premise that you measure inside your own process. You hook in right before a request goes out and right after it comes back, and record that span. That works perfectly for a self-hosted agent loop.

But with Managed Agents, the body of the loop lives inside the sandbox. My code only launches it, receives the stream, and takes the final metadata. In other words, I have exactly two points where measurement is possible: the flow of events visible from outside, and the metadata available after it ends. I cannot insert my own tracer into each step inside the sandbox.

Accept that constraint and the design direction settles itself. Give up on peering inside; instead, record every externally visible event without dropping any, and reconstruct the run from that accumulation. You move the subject of observation from "the process" to "the ledger of events."

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN
✦A run-ledger KV schema and append-per-step implementation that reconstructs a run purely from stream events and final metadata, assuming you can never enter the sandbox
✦How to set the silent-stall threshold from idle-since-last-progress, plus the measured drop in my nightly runs from ~21 minutes to ~80 seconds to detection
✦Code that normalizes failures into seven classes and a table that separates expected failures from the ones you actually need to chase
Secure payment via Stripe · Cancel anytime
✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

or
Unlock all articles with Membership →
Share

Thank You for Reading

Gemini Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.

  • ✦Copy-paste ready implementation code
  • ✦New advanced guides published daily
  • ✦$5/mo or $10 for lifetime access
View Membership →

Related Articles

◈ API / SDK2026-08-04
Retrying a Retired Model Never Showed Up in Success Latency
With the image generation models shutting down soon, I rebuilt a mock server to see what my retry layer actually does when it hits a retired model ID. The damage landed in wall time and queue wait, and never touched the metric I was watching.
◈ API / SDK2026-08-02
Measuring a Guard in environment hooks: 46 Microseconds to Decide, 23 Milliseconds to Start
A record of building a destructive-command guard for Managed Agents environment hooks. A regex denylist let 9 of 20 dangerous commands through; argv parsing reached 100 percent detection. The startup cost turned out to be 500 times the decision cost.
◈ API / SDK2026-05-04
Solving Gemini API Cold Starts — Production-Grade Startup Optimization for Cloud Run, Lambda, and Workers
When you put Gemini API on serverless, the first request takes six seconds. This guide breaks down where the time goes and shows concrete startup-optimization patterns for Cloud Run, AWS Lambda, and Cloudflare Workers — with real numbers, runnable code, and cost trade-off advice.