◈ API / SDK/2026-07-14Advanced

When a Batch Job Sat in RUNNING for Half a Day: Field Notes on Catching Stalls Early with Per-State Dwell Budgets and Record Reconciliation

When a Gemini Batch job stalls quietly under the shadow of the 24-hour SLA, per-state dwell-time budgets and submitted-vs-completed record reconciliation let you name the stall early. Field notes with real operational numbers.

Gemini API²⁰⁹ Batch API⁵ SLO² Observability⁴ Operations¹⁰

✦ Premium Article

I opened the nightly batch dashboard first thing in the morning and the job was frozen in RUNNING. Submitted at 22:00 the night before. Eleven hours had passed. But the Batch API SLA promises completion "within 24 hours," and our monitoring was wired to that 24-hour threshold, so no alert had fired.

Which meant thirteen more hours of not knowing whether the job was alive or dead. Delivery was at noon. It was not going to make it.

The lesson that day was blunt: monitoring keyed to the SLA only tells you about a stall after it's too late to act. What I needed was a way to name an unusual dwell against the usual baseline, early. This note records the approach I landed on for the indie developer pipeline I run — measuring how long jobs sit in each state, drawing a budget from that, and pulling out the jobs that drift past it — along with the numbers from actually running it.

Why an SLA threshold is always too late

The 24-hour SLA is an upper bound: "we will finish within this." In practice, a healthy job finishes in minutes to tens of minutes. That gap is the trap.

If your population of healthy jobs finishes in 20 minutes and you set the threshold at 24 hours, you can only detect a stalled job "after it runs 70x slower than normal." A monitor that stays silent at 90 minutes — or three hours — when the baseline is 20 minutes is not really monitoring anything.

So reframe it. The SLA is a contract with the business, not an anomaly threshold. The anomaly threshold should come from your own measured distribution: how long this job normally sits in each state.

Record dwell time on every state transition

First, record how long a job stays in each state. Time spent in QUEUED, time spent in RUNNING — write it to a ledger on each transition. It piggybacks on polling you're already doing, so the added cost is negligible.

# What this records:
# On each poll, observe the state; the moment it changes,
# append "how many seconds it stayed in the previous state" to a ledger.
import time, json
from google import genai
 
client = genai.Client(api_key="YOUR_API_KEY")
 
TERMINAL = {"BATCH_STATE_SUCCEEDED", "BATCH_STATE_FAILED", "BATCH_STATE_CANCELLED"}
 
def track_dwell(name: str, ledger_path: str, poll: int = 30):
    prev_state = None
    entered_at = time.time()
    while True:
        job = client.batches.get(name=name)
        state = job.state.name
        now = time.time()
        if prev_state is None:
            prev_state, entered_at = state, now
        elif state != prev_state:
            record = {
                "name": name,
                "from_state": prev_state,
                "dwell_sec": round(now - entered_at, 1),
                "to_state": state,
                "ts": now,
            }
            with open(ledger_path, "a", encoding="utf-8") as f:
                f.write(json.dumps(record) + "\n")
            prev_state, entered_at = state, now
        if state in TERMINAL:
            return
        time.sleep(poll)

One thing that matters here: keep state-transition logs out of any sampling. They're low-volume, so if they get swept into generic sampling, the exact line you wanted drops and you're left with "no record." Put Batch logs in an unconditionally-retained category.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦Why a 24-hour SLA threshold detects stalls far too late, and how per-state p95 dwell budgets surface them early

✦Code that reconciles submitted vs completed record counts to count rows that vanished silently

✦Measured lead-time improvement after moving from SLA thresholds to budget-based detection

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

Build a per-state budget from past jobs

Once the ledger holds a few dozen jobs, compute the dwell-time distribution per state. Use percentiles, not the mean — a single huge outlier skews the mean instantly. Here I take p50 and p95, and anchor the budget on p95.

# What this computes:
# From the ledger, aggregate per-state dwell percentiles (p50/p95)
# to derive a budget: "stay longer than this in that state = anomaly."
import json, statistics
from collections import defaultdict
from pathlib import Path
 
def build_budget(ledger_path: str):
    dwell = defaultdict(list)
    for line in Path(ledger_path).read_text(encoding="utf-8").splitlines():
        r = json.loads(line)
        dwell[r["from_state"]].append(r["dwell_sec"])
    budget = {}
    for state, xs in dwell.items():
        xs.sort()
        p50 = statistics.median(xs)
        p95 = xs[max(0, int(len(xs) * 0.95) - 1)]
        budget[state] = {"p50": round(p50, 1), "p95": round(p95, 1), "n": len(xs)}
    return budget
 
# Example output:
# {
#   "BATCH_STATE_QUEUED":  {"p50": 45.0,  "p95": 210.0, "n": 63},
#   "BATCH_STATE_RUNNING": {"p50": 980.0, "p95": 2600.0, "n": 63}
# }

In this example, RUNNING had a p95 of about 43 minutes. So "flag it past 43 minutes in RUNNING, warn at double that" is a threshold far closer to reality than 24 hours. The budget stabilizes as the sample grows, so I refresh it weekly from the trailing 30 days.

Name over-budget jobs early

With a budget in hand, have the monitoring loop consult it and pull only the jobs that exceed it. The key is to treat "over budget" in stages: past p95 is observation, past 2x p95 is a warning, and 80% of the SLA is a forced fallback — different granularities.

# What this decides:
# Compare current dwell against the per-state budget and return
# a severity graded by how far over budget it is.
def classify(state: str, dwell_sec: float, budget: dict) -> str:
    b = budget.get(state)
    if not b:
        return "unknown"
    if dwell_sec <= b["p95"]:
        return "ok"
    if dwell_sec <= b["p95"] * 2:
        return "watch"     # drifting off baseline; log it
    return "alert"         # clearly stalled; page a human
 
SLA_HARD = 24 * 60 * 60
SLA_TRIGGER = int(SLA_HARD * 0.8)   # forced fallback at 19.2 hours
 
def should_fallback(total_elapsed: float) -> bool:
    return total_elapsed >= SLA_TRIGGER

Logging at the watch stage lets you find precursors ahead of time rather than after the fact — patterns like "before every stall, QUEUED ran long." Most stalls don't stop abruptly; they drift off the baseline gradually.

Count "stalls" and "silent drops" as separate problems

Dwell monitoring answers "is it moving?" But there's a second Batch hazard: the job reaches SUCCEEDED, yet some of the rows you submitted quietly go missing from the results. Watching state alone can never catch this.

The fix is plain. Remember how many rows you submitted, and after completion, reconcile against the row count in the results.

# What this reconciles:
# Match submitted record count against succeeded/failed counts in the
# results, and count the "vanished rows" that appear nowhere.
def reconcile(submitted: int, succeeded: int, failed: int) -> dict:
    accounted = succeeded + failed
    missing = submitted - accounted
    return {
        "submitted": submitted,
        "succeeded": succeeded,
        "failed": failed,
        "missing": missing,
        "coverage": round(accounted / submitted, 4) if submitted else 0.0,
    }
 
# Example: {"submitted": 5000, "succeeded": 4980, "failed": 6,
#           "missing": 14, "coverage": 0.9988}

Any day missing is non-zero, I investigate — even if the job is marked successful. Usually it traced back to a trailing blank line in the input JSONL, or a stray newline inside a row, so that single row was never accepted and simply disappeared. Logging coverage every run lets you catch the quiet shortfall of "it said success but only 99.88% actually processed" as a number.

Measured: what changed after moving to budget-based detection

Here is how our first response to Batch incidents changed between the 24-hour-SLA era and after switching to per-state p95 budgets. Same pipeline, roughly three months of data.

Metric	SLA threshold only	Per-state budget + reconcile
Mean lead time to detect a stall	~6.5 hours	~22 minutes
Share of incidents that hit delivery	~34%	~7%
Silent drops (missing>0) detected	Effectively never	Same day
False pages per month	0	2–3

False positives went up — inevitable when you tighten the budget. But by splitting watch from alert and paging only on alert, overnight wake-ups settled at two or three a month. The watch drift flows quietly to logs and gets reviewed in the morning. That two-tier setup is what reconciled sensitivity with quiet.

Practical notes for putting it in production

Don't hardcode the budget; redraw it weekly from measurements. As models turn over and input volume shifts, the RUNNING baseline drifts silently. Measure this month with last month's budget and the threshold hollows out.

And keep the fallback path independent of budget monitoring, firing unconditionally at 80% of the SLA. Budget monitoring exists to notice early; the SLA fallback is the last line of defense. Different roles — so strengthening one is no reason to drop the other. Rerouting to on-demand generate_content costs more, but weighed against a blown deadline, it's insurance worth carrying.

If you want to revisit the happy path first, running things in bulk with the Gemini Batch API pairs well with this and makes it clearer where to slot the monitoring in.

From watching state alone to counting how long a job sits in each state — a small switch, but the number of mornings that started with a jolt dropped noticeably. If you're wrestling with your own nightly batch, I hope this helps with tomorrow's next move. Thanks for reading.

Thank You for Reading

Gemini Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.