Retiring the Midnight Polling Loop — Rebuilding My Gemini Batch Monitoring Around Webhooks

Registering the static webhook

Registration is a few lines — with one irreversible detail.

from google import genai
 
client = genai.Client()
 
webhook = client.webhooks.create(
    name="NightlyBatchWebhook",
    subscribed_events=["batch.succeeded", "batch.failed", "batch.expired"],
    uri="https://my-api.example.com/gemini-callback",
)
 
# The signing secret is returned only in this response
print(webhook.new_signing_secret)

new_signing_secret is returned exactly once, at creation time. I failed to save it on my first attempt and had to call rotate_signing_secret. Rotation lets you choose between revoking old secrets immediately or after a 24-hour grace period; in production, REVOKE_PREVIOUS_SECRETS_AFTER_H24 gives you a safe overlap window.

Subscribing to batch.expired is deliberate. The Batch API expires jobs that aren't processed within 24 hours, and in the polling era I often didn't notice until the next morning. Having expiry arrive through the same pipe as success — just another kind of completion — is a real operational win.

The receiver — don't skip signature verification

I wrote the receiver in Flask. Gemini webhooks follow the Standard Webhooks specification, so verification can be delegated to the standardwebhooks library.

# pip install flask standardwebhooks
import os
import queue
import threading
from flask import Flask, request, jsonify
from standardwebhooks.webhooks import Webhook, WebhookVerificationError
 
app = Flask(__name__)
SIGNING_SECRET = os.environ["WEBHOOK_SIGNING_SECRET"]
 
# Push heavy work to a worker thread; respond immediately
work_queue: "queue.Queue[dict]" = queue.Queue()
seen_ids: set[str] = set()  # webhook-id dedup (use a TTL store in production)
 
@app.route("/gemini-callback", methods=["POST"])
def gemini_callback():
    payload = request.get_data(as_text=True)
    try:
        wh = Webhook(SIGNING_SECRET)
        event = wh.verify(payload, request.headers)
    except WebhookVerificationError:
        return jsonify({"error": "invalid signature"}), 400
 
    delivery_id = request.headers.get("webhook-id", "")
    if delivery_id in seen_ids:
        return jsonify({"status": "duplicate"}), 200
    seen_ids.add(delivery_id)
 
    work_queue.put(event)  # parsing and downloads happen in the worker
    return jsonify({"status": "received"}), 200
 
def worker():
    while True:
        event = work_queue.get()
        if event.get("type") == "batch.succeeded":
            uri = event["data"]["output_file_uri"]
            download_and_ingest(uri)  # fetch and ingest the results file
        elif event.get("type") in ("batch.failed", "batch.expired"):
            notify_failure(event["data"])
 
threading.Thread(target=worker, daemon=True).start()

Three operational requirements are baked into this code.

1. Verify the signature first. The webhook-signature, webhook-id, and webhook-timestamp headers are checked together by standardwebhooks. Deliveries with timestamps older than 5 minutes are rejected as potential replays
2. Return 2xx immediately. A slow response pushes Gemini into its retry cycle. Never do heavy work inside the handler
3. Deduplicate on webhook-id. Delivery is at-least-once. Assume every notification can arrive twice and keep ingestion idempotent

Things the documentation doesn't tell you

What follows comes from roughly three weeks of running this in production.

The payload is thin. A notification carries output_file_uri and counts — not the results themselves. Going event-driven removes the polling loop and nothing else; all the code that fetches and parses results stays. Budgeting for that from the start keeps the design honest.

Local development is quietly painful. To test with real signatures, the most reliable path was tunneling into my dev machine (cloudflared or similar) and receiving live events. Registering a second static webhook pointed at the dev environment beat hand-forging signature headers every time.

I kept a fallback poll. At-least-once delivery doesn't cover your receiver being down or DNS misbehaving. My insurance: if no notification has arrived 6 hours after submission, do a single GET. Worst case that's 3 calls a night instead of 1,080 — an acceptable premium.

For the broader question of surviving API outages, I've written separately about keeping a nightly batch alive through a Gemini API outage.

The migration checklist

The order I actually followed. The point is the overlap period — never cut over in one move.

1. Implement the receiver and test the three essentials: signature verification, deduplication, immediate 2xx
2. Register the static webhook for batch.succeeded / batch.failed / batch.expired and store the secret in environment variables
3. Run webhooks in parallel with polling for one week, cross-checking that both detect the same completions
4. Relax the polling interval from 60 seconds to the 6-hour insurance level
5. Only after the overlap log shows zero misses, delete the cron watcher

During the overlap, one notification arrived while my receiver was mid-restart. Gemini's exponential backoff retried for up to 24 hours, so nothing was lost — but that night is what convinced me to keep the fallback poll permanently.

Results, measured

• Status-check GETs: ~1,080/night → 0 in normal operation (at most 3 from the fallback)
• Lag from completion to ingestion start: ~30 seconds average → a few seconds
• Monitoring code: ~180 lines → ~110 lines (the polling loop and backoff control simply vanished)
• Silent watcher deaths: zero across three weeks of parallel running

The number that matters least is the one I feel most: I no longer reason about whether a cron process is alive. If the event doesn't come, the insurance catches it. A simpler structure sleeps better at night — and so do I.

A first step

Start small: attach a dynamic webhook to a single ad-hoc job and route it with user_metadata. Static registration affects the whole project, so there is no harm in building intuition on a throwaway job first.

If you run nightly batches of your own, I hope this record saves you an early morning or two.