The Morning Gemini Generated Fine but the Publish Crashed — A 'Generation Outbox' So Expensive Output Is Never Lost

A single outbox table

The store can be anything. Since I run this solo, I started with SQLite, which lives in one file. On Cloudflare Workers you can swap in D1 or KV.

CREATE TABLE IF NOT EXISTS outbox (
  fingerprint   TEXT PRIMARY KEY,   -- fingerprint of the generation request (idempotency key)
  target        TEXT NOT NULL,      -- where to publish (article slug, site name, ...)
  payload       TEXT NOT NULL,      -- the Gemini output itself
  tokens        INTEGER,            -- measured token count (for cost tracking)
  status        TEXT NOT NULL DEFAULT 'pending',  -- pending / published
  created_at    REAL,
  published_at  REAL
);

The key point is making fingerprint the primary key. As we will see, it is the single axis that suppresses both duplicate generation and duplicate publishing. The payload holds the output verbatim. The instant it lands there, the output becomes an asset you never need to buy again.

Generation phase — the job ends once output is in the box

Here is the generation side. It computes the fingerprint, skips generation if the box already has it, and otherwise calls Gemini and writes the result.

import hashlib
import json
import time
import sqlite3
from google import genai
 
client = genai.Client()
db = sqlite3.connect("outbox.db")
 
 
def fingerprint(model: str, prompt: str, config: dict) -> str:
    """Hash with sorted keys so identical inputs map to the same fingerprint."""
    payload = json.dumps(
        {"model": model, "prompt": prompt, "config": config},
        sort_keys=True,
        ensure_ascii=False,
    )
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()
 
 
def generate_into_outbox(model: str, prompt: str, config: dict, target: str) -> str:
    fp = fingerprint(model, prompt, config)
 
    row = db.execute(
        "SELECT status FROM outbox WHERE fingerprint = ?", (fp,)
    ).fetchone()
    if row is not None:
        # Already generated. Skip regeneration; hand off to the publish phase.
        return fp
 
    resp = client.models.generate_content(
        model=model,
        contents=prompt,
        config=config,
    )
 
    db.execute(
        "INSERT INTO outbox "
        "(fingerprint, target, payload, tokens, status, created_at) "
        "VALUES (?, ?, ?, ?, 'pending', ?)",
        (
            fp,
            target,
            resp.text,
            resp.usage_metadata.total_token_count,
            time.time(),
        ),
    )
    db.commit()
    return fp

Once commit() returns, the output is on disk. If the process dies after this line, there is no need to call Gemini again. The generation phase can safely let go here.

Publish phase — an independent drain that empties the box

Publishing runs as a separate drain process, on its own schedule. It pulls pending rows, publishes them, and flips them to published on success.

def drain_outbox(publish_fn) -> int:
    """Publish pending output. Safe to call any number of times."""
    rows = db.execute(
        "SELECT fingerprint, target, payload FROM outbox "
        "WHERE status = 'pending'"
    ).fetchall()
 
    done = 0
    for fp, target, payload in rows:
        # Pass fingerprint to the publisher as the idempotency key.
        # If the same key arrives twice, the publisher collapses it to one.
        ok = publish_fn(target=target, body=payload, idempotency_key=fp)
        if not ok:
            # Could not publish this time. Leave it pending for the next run.
            continue
        db.execute(
            "UPDATE outbox SET status = 'published', published_at = ? "
            "WHERE fingerprint = ?",
            (time.time(), fp),
        )
        db.commit()
        done += 1
    return done

publish_fn can be a git push, a CMS posting API, or a webhook. What matters is that it always accepts an idempotency_key and rejects duplicates on the receiving side. Many posting APIs offer an idempotency-key header; for a homegrown target, a single "have I already published this key?" check is enough.

Verify it survives a crash at every point

Whether the design is correct becomes clear when you walk each crash point one by one.

Crash before generation. The box is empty. The retry simply regenerates. Zero loss.

Crash right after generate_content but before commit(). The output was still only in memory. This is the one window where re-paying is unavoidable — which is exactly why you keep this window as short as possible and write immediately after generation.

Crash after commit() but before publishing. This is the exact point I hit during the June outage. The output is sitting in the box as pending. The next drain picks it up and publishes it, so generation cost does not rise at all.

Crash after publishing but before flipping to published. The next drain tries to publish the same row again, but thanks to the idempotency_key the target rejects the second attempt. Readers never see a double post.

Across these four points, "regenerate the expensive output" is confined to the tiny second window. That confinement is what the outbox buys you.

A bonus that mattered during the outage

What I felt during the June incident is that splitting generation from publishing also raises resilience to the outage itself.

Even when the publish target (GitHub) was flaky, as long as Gemini responded I could keep generating, and every output piled up in the box. Once the target recovered, a single drain flushed all the accumulated pending rows at once. Conversely, during the hours Gemini was returning error 1076, I gave up only on generation and calmly kept publishing whatever was already in the box.

The outage-recovery design itself is covered in Building a Nightly Batch That Does Not Stop, but the outbox is its foundation. When one side falls over, the other keeps moving through the box. That separation kept the whole pipeline alive that day.

Pitfalls I hit in practice

After running it for a while, a few things stood out.

The scariest pitfall is pending rows piling up unnoticed. If the target stays unhealthy, the box keeps filling. I recommend folding "count of pending rows older than N hours by created_at" into your morning ops digest, with a threshold that surfaces it.

The fingerprint must include every setting that affects the output. Not just the model and prompt, but temperature and response_schema too — otherwise a regeneration with only the config changed gets misread as "the same thing" and skipped. I settled on hashing the entire config.

Decide when to delete published rows. Because I want tokens for cost tracking, a separate job only sweeps rows that have been published for more than 30 days.

Choosing by scale

To start small, one SQLite file is plenty. My own solo setup drove the "generated but not published" loss to zero with exactly this.

On edge runtimes like Cloudflare Workers, swap the store for D1 and the backlog monitoring for Workers Analytics, and you get the same design without owning a server.

If you add more publish targets and want stronger delivery guarantees, leaning on a dedicated durable-workflow platform is an option; I lay out that decision in Durable Workflows with the Gemini API. But before reaching for a big machine, plug the costliest failure with a one-table outbox first. For indie work, that is where I find the cost-to-benefit ratio is highest.

Thanks for reading. Once you have paid for a generation, you want to see that output through to a single, reliable publish. Since I started thinking that way, my pipeline has been a little quieter even on the morning of an outage.