Recording Provenance for Gemini Output — Designing for Reproducibility and Audit

Wrapping the call so provenance is never optional

If you treat provenance as something to "add later," you will forget it. The reliable approach is to wrap the generation call itself so that output and provenance always come back together.

from google import genai
from google.genai import types
 
client = genai.Client()
SDK_VERSION = "google-genai 1.x"
MODEL = "gemini-2.5-flash"
MODEL_GENERATION = "2.5"
 
def generate_with_provenance(
    template_id: str,
    variables: dict,
    prompt_text: str,
    *,
    temperature: float = 0.0,
    seed: int | None = 7,
    thinking_budget: int | None = None,
):
    config = types.GenerateContentConfig(temperature=temperature, seed=seed)
    response = client.models.generate_content(
        model=MODEL, contents=prompt_text, config=config
    )
    output = response.text
 
    prov = Provenance(
        model=MODEL,
        model_generation=MODEL_GENERATION,
        prompt_template_id=template_id,
        prompt_hash=sha256_short(canonical_prompt(template_id, variables)),
        temperature=temperature,
        thinking_budget=thinking_budget,
        seed=seed,
        input_hash=sha256_short(prompt_text),
        output_hash=sha256_short(output),
        sdk_version=SDK_VERSION,
        created_at=time.time(),
    )
    return output, prov

The caller simply receives output, prov = generate_with_provenance(...). So that a provenance-less path can be caught in review, I keep a strict rule — even as a team of one — that no code touches the raw API anywhere except inside this wrapper.

Even at temperature 0, you can't fully reproduce it

This is the most misunderstood point. People assume that fixing temperature=0 and a seed yields the identical output every time. On Gemini, it does not. Temperature 0 strengthens the tendency to pick the highest-probability token, but it guarantees no determinism. Backend model updates, floating-point non-determinism, and hardware differences all let the same input drift.

In my own wallpaper-classification measurements, even at temperature 0 with a fixed seed, sending the same image 100 times produced a few results in a different category. The agreement rate sat around 97% — roughly 3% drifts. That is exactly why a provenance design must separate "what can be reproduced."

Reproducible are the generation conditions. Which model, which prompt, which settings — with provenance, you can rebuild them exactly. Not reproducible is bit-for-bit output equality. Give that up, and instead use output_hash as a comparison key for "is today's output the same as the one from back then?" If the hash matches, it is identical; if not, use the other provenance fields to separate "did it drift, or did a condition change?" Framing reproducibility as "condition reproduction" plus "output comparison" keeps the design from wobbling.

Where to store provenance — embedded, sidecar, or database

There are three main options for storage, each with a different character. Here are situational recommendations.

The first is embedding it into the output: add a _provenance key inside the generated JSON, so output and provenance never physically separate. The downside is that it pollutes the output schema, so it is a poor fit when you want to keep it apart from what you return to users.

The second is a sidecar: place provenance in a separate file or record from the output body, linked by a shared ID. This is what my wallpaper classification uses. It avoids polluting the output schema and makes it easy to later compress or delete provenance on its own.

The third is centralizing into a dedicated table. If you need cross-cutting queries — "pull only generations with a specific prompt hash from last month" — this is the strongest option.

import sqlite3
 
def save_provenance(db, record_id: str, output: str, prov: Provenance):
    db.execute(
        "INSERT INTO generations(id, output, prov) VALUES(?, ?, ?)",
        (record_id, output, json.dumps(asdict(prov), ensure_ascii=False)),
    )
    db.commit()
 
def replay_conditions(db, record_id: str) -> dict:
    # Pull the provenance needed to rebuild the original generation conditions
    row = db.execute(
        "SELECT prov FROM generations WHERE id = ?", (record_id,)
    ).fetchone()
    prov = json.loads(row[0])
    return {
        "model": prov["model"],
        "prompt_template_id": prov["prompt_template_id"],
        "prompt_hash": prov["prompt_hash"],
        "temperature": prov["temperature"],
        "seed": prov["seed"],
    }

At indie scale, I'd recommend starting with a sidecar and moving to a dedicated table only once the need for cross-cutting investigation appears. There is no need to build heavy infrastructure up front.

The moment audit paid off — reproducing a misfire in three minutes

Back to that misclassification report. After provenance was in place, a similar incident played out completely differently. From the reported image's ID I pulled the provenance, rebuilt the conditions with replay_conditions, and re-ran with the same prompt template and settings. The whole thing took about three minutes.

When I reproduced it, the provenance's template ID immediately revealed that the misclassification was not model drift — it came from dropping a category description when I bumped the prompt template from v2 to v3. Without provenance, I would have spent hours staring at the output, wondering whether the model or the prompt was at fault.

The speed of that investigation matters most for the features tied directly to AdMob revenue. Classification quality shapes the experience, and the experience shapes retention. When something breaks in production, the time it takes to isolate the cause is the scarcest resource a solo developer has. I treat provenance as an investment to buy that time back.

Pitfalls that are easy to hit

Finally, here are the points where adding provenance to production tends to trip people up, each paired with how to handle it.

1. Not stripping PII before hashing. Hashing the full input does not let anyone recover PII, but if you dump the prompt body into logs alongside it, it leaks. Keep only the hash in provenance, and separate the body logs.
2. Changing the prompt body while leaving the template ID untouched. Do this and a different prompt carries the same hash — your provenance now lies. Avoid it with the discipline of always bumping the version on a body change.
3. Not recording the SDK version. When an SDK update shifts a default, behavior changes silently. Keeping sdk_version lets you suspect environment drift before you suspect the model.
4. Recording provenance only on success. Errors and empty outputs are exactly what you want to reproduce. Stamping provenance on failures too raises the resolution of your investigations by a level.

Every one of these is preventable with a single line of care in the initial design. Try to bolt it on afterward, and your past provenance is locked in incomplete.

The next step

Pick the single generation path that causes the most trouble, and route only that one through a thin wrapper like generate_with_provenance. You do not need to replace every path at once. Once you have felt, on a single path, what it is like to "rebuild the original conditions in three minutes," the value of recording provenance becomes a concrete, tangible thing.

If you are running generative AI in production too, I hope this helps a little with isolating causes when something goes wrong.