Hung Jobs Fail Quietly — Wrapping a Scheduled Gemini Call With a Timeout, an Empty-Output Check, and a Lock

When a scheduled job calls gemini -p "...", the first sign of trouble is rarely an error. It is a morning when nothing arrives. No error message, an empty log, and a process that is still sitting there.

A command waiting on a response will not end on its own. My current habit is to wrap the call in an outer layer that ends it from the outside.

Below is that wrapper, about 40 lines of shell, and what I measured while testing it. I did not use the real Gemini for the measurements. I used stand-in commands that hang, return nothing, or fail, so how often Gemini itself stalls is outside this note. Everything here is what you can see from the outside: exit codes and seconds.

Three things to guard against

All three can happen with any command, not just Gemini, so the command to run is swappable through an environment variable.

The full script

#!/usr/bin/env bash
# gemini-guard.sh — usage: gemini-guard.sh prompt.txt out.md
set -u
 
PROMPT_FILE="${1:?usage: gemini-guard.sh prompt.txt out.md}"
OUT="${2:?usage: gemini-guard.sh prompt.txt out.md}"
LIMIT="${GUARD_LIMIT:-120}"          # seconds; stop the job after this
GRACE="${GUARD_GRACE:-10}"           # seconds between the stop request and a forced kill
LOCK="${GUARD_LOCK:-/tmp/gemini-guard.lock}"
LOG="${GUARD_LOG:-$HOME/gemini-guard.jsonl}"
CMD="${GUARD_CMD:-gemini}"
 
log() {  # log <status> <rc> <seconds> <bytes>
  printf '{"ts":"%s","status":"%s","rc":%s,"sec":%s,"bytes":%s}\n' \
    "$(date -u +%FT%TZ)" "$1" "$2" "$3" "$4" >> "$LOG"
}
 
# If the previous run is still going, give up instead of piling on
exec 9>"$LOCK"
if ! flock -n 9; then
  log skipped_locked 0 0 0
  exit 75
fi
 
start=$(date +%s)
timeout --kill-after="$GRACE" "$LIMIT" "$CMD" -p "$(cat "$PROMPT_FILE")" \
  > "$OUT.tmp" 2> "$OUT.err"
rc=$?
sec=$(( $(date +%s) - start ))
bytes=$(wc -c < "$OUT.tmp" | tr -d ' ')
 
case "$rc" in
  0)
    if [ "$bytes" -eq 0 ]; then
      log empty_output "$rc" "$sec" "$bytes"; rm -f "$OUT.tmp"; exit 3
    fi
    mv "$OUT.tmp" "$OUT"            # only publish a finished result
    log ok "$rc" "$sec" "$bytes"; exit 0 ;;
  124) log timeout "$rc" "$sec" "$bytes"; rm -f "$OUT.tmp"; exit 124 ;;
  137) log killed  "$rc" "$sec" "$bytes"; rm -f "$OUT.tmp"; exit 137 ;;
  *)   log failed  "$rc" "$sec" "$bytes"; rm -f "$OUT.tmp"; exit "$rc" ;;
esac

A few notes on why it looks this way.

Writing to .tmp and renaming with mv keeps a half-written result away from whatever reads the file next. Within one directory the rename is effectively instant.

The exit codes differ by cause: 124 when timeout ended the job, 137 when it ignored the stop request and was force-killed, 75 when the run was skipped as a late starter. A cron mail or a monitor can tell them apart at a glance.

What the stand-ins showed

I shrank the limit to 3 seconds and the grace period to 2, then ran five stand-ins.

The 5 seconds for the TERM-ignoring case is the 3-second limit plus the 2-second grace. Without --kill-after, that stand-in would have stayed until its 60-second wait ended. Programs that swallow TERM are not rare, so I keep the grace period.

I also checked overlap. I started a hanging stand-in, then a second run one second later. The second exited immediately with 75, and the first ended at 3 seconds with 124. The log shows skipped_locked followed by timeout.

--foreground changes how far the stop reaches

The part I did not expect was what happens to grandchildren. I used a stand-in script that calls sleep 77 internally, stopped it two ways, and counted what remained (GNU coreutils 8.32).

By default, timeout runs the command in its own process group and stops the whole group when time is up. With --foreground it does not separate the group, so only the direct child is stopped and a grandchild stays alive. Calling setsid inside the command left a process behind for the same reason.

The wrapper does not use --foreground, so that cleanup works even if gemini spawns helpers. There are situations at an interactive terminal where --foreground is needed, so choose by use.

Putting it in cron

# 06:10 daily. If runs overlap, the wrapper skips the late one
10 6 * * * GUARD_LIMIT=180 /home/me/bin/gemini-guard.sh /home/me/jobs/morning.txt /home/me/out/morning.md

One line for the morning check makes it easier.

tail -n 3 ~/gemini-guard.jsonl

If anything other than ok shows up, standard error is saved in out/morning.md.err. I clear the .err files from successful runs in batches when I remember to.

macOS has no timeout by default. Install coreutils with Homebrew and use gtimeout instead. flock is also absent, so swap in another lock, such as a simple mkdir-based one.

Where the wrapper stops helping

The wrapper tells you that a job stopped, not why. Whether the wait was on Gemini, the network, or an oversized prompt has to be inferred from the .err file and the trend in elapsed seconds.

Choosing the limit has no single right answer either. I start at three to four times the usual runtime and tighten it once enough ok entries have piled up. Starting too tight kills normal, long runs.

End what will not end from the outside first, and record that it did not end. Fixing the inside can wait until the log points at a cause.

For related reading, rebuilding a CLI-dependent scheduled job is covered in Keeping Nightly Batches Alive After the Gemini CLI Stops Responding: A google-genai SDK Fallback, and not losing results from fire-and-forget runs is in Fire-and-Forget on a Cron That Never Loses a Result: Reclaiming Gemini Background Executions with a Submission Ledger.