The Nightly Batch Sat Stalled Until Morning — Splitting Retryable Failures by gemini-api Exit Code

As an indie developer, I wanted to queue 20 wallpaper images overnight and pick from them in the morning. The worst outcome is waking up to a batch that stopped at the third one.

Without knowing why it stopped, you cannot decide what to do. A network blip is fixed by running it again. An expired API key fails at the same place every time.

The 10/27 release notes added a command called gemini-api. The official page describes it as a stateless, zero-dependency beta client for AI coding agents, scripts and CI. It is a different tool from the interactive Gemini CLI: it simply calls the API, one command at a time.

This note is about one thing only: the exit codes. Not a tour of the commands, but where to stop, where to wait and where to skip when the tool sits inside a nightly job.

Everything below is a design note based on the official page, not a record of running this for months. It is a beta, so please try it with --dry-run on your own machine first.

The official page defines only four exit codes

The table on the official page lists these four.

What stands out is that code 1 bundles three different kinds of failure. A network blip, a rate limit and a failed background job are mixed together, and only some of them heal by waiting.

Codes 2 and 3 are the opposite. Waiting changes nothing, because either the input is broken or the key is rejected.

The rule: wait only for failures that waiting can change, and stop early on the rest.

The wrapper

Here is that rule written as a script. Each item goes through run_with_retry, which branches on the exit code.

#!/usr/bin/env bash
# nightly-gemini.sh — decide whether to retry from the gemini-api exit code
set -u
 
MAX_ATTEMPTS=3
LOG_DIR="${LOG_DIR:-./logs}"
mkdir -p "$LOG_DIR" ./out
 
# Returns: 0=ok, 2=bad input (skip), 3=auth problem (stop everything), 1=retries exhausted
run_with_retry() {
  local name="$1"; shift
  local attempt=1 code wait_sec
 
  while :; do
    # --agent-mode: structured JSON errors on stderr. Keep one file per attempt
    "$@" --agent-mode 2> "$LOG_DIR/${name}.attempt${attempt}.err"
    code=$?
 
    case "$code" in
      0) return 0 ;;
      2) echo "[$name] input validation error (exit 2). Skipping, no retry" >&2; return 2 ;;
      3) echo "[$name] auth error (exit 3). Stopping the batch" >&2; return 3 ;;
      1)
        if [ "$attempt" -ge "$MAX_ATTEMPTS" ]; then
          echo "[$name] failed after ${MAX_ATTEMPTS} attempts (exit 1)" >&2
          return 1
        fi
        wait_sec=$(( 30 * attempt * attempt ))   # 30s, then 120s
        echo "[$name] exit 1. Waiting ${wait_sec}s before retry (${attempt}/${MAX_ATTEMPTS})" >&2
        sleep "$wait_sec"
        attempt=$(( attempt + 1 ))
        ;;
      *) echo "[$name] unexpected exit code $code" >&2; return "$code" ;;
    esac
  done
}
 
skipped=0; failed=0; line=0
while IFS= read -r prompt; do
  line=$(( line + 1 ))
  n=$(printf '%03d' "$line")   # input line number keeps log and output names stable
  run_with_retry "wp-$n" gemini-api image "$prompt" --out "./out/wp-$n.jpg"
  case $? in
    0) ;;
    2) skipped=$(( skipped + 1 )) ;;
    3) echo "Authentication failed; aborting" >&2; exit 3 ;;
    *) failed=$(( failed + 1 )) ;;
  esac
done < prompts.txt
 
echo "Done: skipped=${skipped} failed=${failed}"
[ "$failed" -eq 0 ] && [ "$skipped" -eq 0 ]

I build file names from the input line number so that, even when some lines fail, the morning question "which line was this image?" has a direct answer.

Logs are split per attempt because the stderr from --agent-mode is the only clue for telling what was inside a code 1 (a blip or a rate limit). The field names of the error JSON are not listed on the official page, so I suggest failing on purpose once and reading the real output before writing any aggregation.

Rerunning can bill you twice: background jobs inside code 1

The wrapper above is meant for short work such as image generation.

Long jobs like the Deep Research example in the docs are different. With --background, the command is documented to return an interaction ID immediately. If you rerun the same command without that ID, a second job starts.

Since code 1 includes "failed background job", a wrapper that sees 1 and reruns the same command may bill you twice. It is safer to assume it can.

For long jobs, I skip the wrapper and do this instead.

1. Start with --background --stream=false and write the returned ID to a file
2. Afterwards ask about that ID with agent status (check the output format on the official page before writing a parser)
3. Use agent cancel <id> when you want it stopped

In short, retry only when you can tell nothing has started. If you hold an ID, switch from rerunning to asking.

Fail the evening before, not in the morning

A code 2 validation error is a poor thing to discover after the night has passed. According to the official page, --dry-run validates inputs and previews the request offline, with credentials redacted.

So I put a preflight in front of the real run that pushes every line through --dry-run only.

# preflight.sh — validate every input line before the real run
bad=0
while IFS= read -r prompt; do
  if ! gemini-api image "$prompt" --out /dev/null --dry-run >/dev/null 2>&1; then
    echo "Validation error: $prompt" >&2
    bad=$(( bad + 1 ))
  fi
done < prompts.txt
[ "$bad" -eq 0 ] || { echo "${bad} line(s) have problems" >&2; exit 2; }

I could not tell from the official page whether --out /dev/null is accepted under --dry-run. If it is not, swap in an ordinary path such as --out ./out/dryrun.jpg.

Two precautions for running a beta overnight

The official page gives two beta cautions.

The first is that breaking changes can arrive even between minor versions. Pin a release tag with GEMINI_VERSION at install time. If the meaning of an exit code changes, the branches above break as they stand.

The second is that generate and agent run need --stream=false when combined with -o json or --jq. If you added -o json inside the wrapper and nothing prints, look here first.

What to do tonight

• Pass authentication through the environment variable GEMINI_API_KEY (never hard-code it in the script; set export GEMINI_API_KEY=YOUR_API_KEY in the calling shell)
• Run preflight.sh on three prompts and confirm the exit code is 0 or 2
• Change one character of the key on purpose, run run_with_retry, and watch it stop immediately on 3

A branch on exit codes can look right when you write it, and you cannot trust it until you have watched it stop with your own eyes. I prefer to make it fail deliberately before leaving it to the night.

If you hit trouble with the interactive CLI instead, see Gemini CLI Won't Start or Authenticate? Where to Look First. And if responses changed because of automatic model routing, The Morning Flash Started Answering Differently, Your Prompt Is Not the First Thing to Check may help.