When a Batch Job Sat in RUNNING for Half a Day: Field Notes on Catching Stalls Early with Per-State Dwell Budgets and Record Reconciliation

Build a per-state budget from past jobs

Once the ledger holds a few dozen jobs, compute the dwell-time distribution per state. Use percentiles, not the mean — a single huge outlier skews the mean instantly. Here I take p50 and p95, and anchor the budget on p95.

# What this computes:
# From the ledger, aggregate per-state dwell percentiles (p50/p95)
# to derive a budget: "stay longer than this in that state = anomaly."
import json, statistics
from collections import defaultdict
from pathlib import Path
 
def build_budget(ledger_path: str):
    dwell = defaultdict(list)
    for line in Path(ledger_path).read_text(encoding="utf-8").splitlines():
        r = json.loads(line)
        dwell[r["from_state"]].append(r["dwell_sec"])
    budget = {}
    for state, xs in dwell.items():
        xs.sort()
        p50 = statistics.median(xs)
        p95 = xs[max(0, int(len(xs) * 0.95) - 1)]
        budget[state] = {"p50": round(p50, 1), "p95": round(p95, 1), "n": len(xs)}
    return budget
 
# Example output:
# {
#   "BATCH_STATE_QUEUED":  {"p50": 45.0,  "p95": 210.0, "n": 63},
#   "BATCH_STATE_RUNNING": {"p50": 980.0, "p95": 2600.0, "n": 63}
# }

In this example, RUNNING had a p95 of about 43 minutes. So "flag it past 43 minutes in RUNNING, warn at double that" is a threshold far closer to reality than 24 hours. The budget stabilizes as the sample grows, so I refresh it weekly from the trailing 30 days.

Name over-budget jobs early

With a budget in hand, have the monitoring loop consult it and pull only the jobs that exceed it. The key is to treat "over budget" in stages: past p95 is observation, past 2x p95 is a warning, and 80% of the SLA is a forced fallback — different granularities.

# What this decides:
# Compare current dwell against the per-state budget and return
# a severity graded by how far over budget it is.
def classify(state: str, dwell_sec: float, budget: dict) -> str:
    b = budget.get(state)
    if not b:
        return "unknown"
    if dwell_sec <= b["p95"]:
        return "ok"
    if dwell_sec <= b["p95"] * 2:
        return "watch"     # drifting off baseline; log it
    return "alert"         # clearly stalled; page a human
 
SLA_HARD = 24 * 60 * 60
SLA_TRIGGER = int(SLA_HARD * 0.8)   # forced fallback at 19.2 hours
 
def should_fallback(total_elapsed: float) -> bool:
    return total_elapsed >= SLA_TRIGGER

Logging at the watch stage lets you find precursors ahead of time rather than after the fact — patterns like "before every stall, QUEUED ran long." Most stalls don't stop abruptly; they drift off the baseline gradually.

Count "stalls" and "silent drops" as separate problems

Dwell monitoring answers "is it moving?" But there's a second Batch hazard: the job reaches SUCCEEDED, yet some of the rows you submitted quietly go missing from the results. Watching state alone can never catch this.

The fix is plain. Remember how many rows you submitted, and after completion, reconcile against the row count in the results.

# What this reconciles:
# Match submitted record count against succeeded/failed counts in the
# results, and count the "vanished rows" that appear nowhere.
def reconcile(submitted: int, succeeded: int, failed: int) -> dict:
    accounted = succeeded + failed
    missing = submitted - accounted
    return {
        "submitted": submitted,
        "succeeded": succeeded,
        "failed": failed,
        "missing": missing,
        "coverage": round(accounted / submitted, 4) if submitted else 0.0,
    }
 
# Example: {"submitted": 5000, "succeeded": 4980, "failed": 6,
#           "missing": 14, "coverage": 0.9988}

Any day missing is non-zero, I investigate — even if the job is marked successful. Usually it traced back to a trailing blank line in the input JSONL, or a stray newline inside a row, so that single row was never accepted and simply disappeared. Logging coverage every run lets you catch the quiet shortfall of "it said success but only 99.88% actually processed" as a number.

Measured: what changed after moving to budget-based detection

Here is how our first response to Batch incidents changed between the 24-hour-SLA era and after switching to per-state p95 budgets. Same pipeline, roughly three months of data.

False positives went up — inevitable when you tighten the budget. But by splitting watch from alert and paging only on alert, overnight wake-ups settled at two or three a month. The watch drift flows quietly to logs and gets reviewed in the morning. That two-tier setup is what reconciled sensitivity with quiet.

Practical notes for putting it in production

Don't hardcode the budget; redraw it weekly from measurements. As models turn over and input volume shifts, the RUNNING baseline drifts silently. Measure this month with last month's budget and the threshold hollows out.

And keep the fallback path independent of budget monitoring, firing unconditionally at 80% of the SLA. Budget monitoring exists to notice early; the SLA fallback is the last line of defense. Different roles — so strengthening one is no reason to drop the other. Rerouting to on-demand generate_content costs more, but weighed against a blown deadline, it's insurance worth carrying.

If you want to revisit the happy path first, running things in bulk with the Gemini Batch API pairs well with this and makes it clearer where to slot the monitoring in.

From watching state alone to counting how long a job sits in each state — a small switch, but the number of mornings that started with a jolt dropped noticeably. If you're wrestling with your own nightly batch, I hope this helps with tomorrow's next move. Thanks for reading.