Tracing Which Prompt Revision Moved Your Quality — Prompt Versioning for a Gemini Pipeline

Stamp the model ID and prompt hash on every generation

Attach the pinned revision ID to the output's metadata, every time. This is the most important difference between before and after.

# before — wording inlined, no provenance in the output
from google import genai
 
client = genai.Client()
 
def draft_article_before(topic: str) -> str:
    prompt = f"Write a careful Japanese article draft on this topic: {topic}"
    resp = client.models.generate_content(
        model="gemini-flash-latest",
        contents=prompt,
    )
    return resp.text  # which wording / which model produced this? unknown later

# after — ship the revision ID and resolved model ID with the output
from google import genai
from prompt_registry import load_prompt
 
client = genai.Client()
 
def draft_article_after(topic: str) -> dict:
    p = load_prompt("article_ja_draft")
    prompt = p["text"].format(topic=topic)
    resp = client.models.generate_content(
        model="gemini-flash-latest",
        contents=prompt,
    )
    # also record what the -latest alias actually resolved to
    resolved = getattr(resp, "model_version", None) or "gemini-flash-latest"
    return {
        "text": resp.text,
        "prompt_id": p["id"],
        "prompt_revision": p["revision"],
        "model": resolved,
        "topic": topic,
    }

Write the prompt_revision and model returned by the after version into every line of your generation log. That alone fixes "which wording and which actual model made this output." Even when model_version isn't present in the response, at least record the model name you requested. Keeping both lets you later see the window where "the wording was the same but only the model moved."

Make the log one JSON line per generation

import json, datetime
 
def append_log(record: dict, log_path="gen_log.jsonl"):
    record["ts"] = datetime.datetime.now(datetime.timezone.utc).isoformat()
    with open(log_path, "a", encoding="utf-8") as f:
        f.write(json.dumps(record, ensure_ascii=False) + "\n")

A one-record-per-line JSONL drops straight into the time-ordered bisect below.

Bisect a quality drop down to the revision boundary

Once provenance is stamped, you can mechanically corner "when did quality drop." Assuming each generation carries a quality score (an automatic eval, or your own 1-to-5 rating), you look for the boundary where the score fell.

On my own data, when I scored the stiffness of sentence endings, the score fell by about 15% across one revision. What my eyes could only call "somehow stiff" showed up as a clear cliff at the revision-hash level.

# bisect_regression.py
import json
 
def load_records(path="gen_log.jsonl"):
    with open(path, encoding="utf-8") as f:
        return [json.loads(line) for line in f if line.strip()]
 
def find_regression_boundary(records, score_key="score", drop=0.10):
    """Return the first revision boundary where mean score fell by >= drop."""
    rows = sorted(records, key=lambda r: r["ts"])
    # aggregate mean score per revision, preserving first-seen order
    seen = []
    agg = {}
    for r in rows:
        rev = r["prompt_revision"]
        if rev not in agg:
            agg[rev] = []
            seen.append(rev)
        agg[rev].append(r.get(score_key, 0.0))
    means = [(rev, sum(agg[rev]) / len(agg[rev])) for rev in seen]
    # return the first adjacent pair whose drop exceeds the threshold
    for (prev_rev, prev_m), (cur_rev, cur_m) in zip(means, means[1:]):
        if prev_m - cur_m >= drop:
            return {"from": prev_rev, "to": cur_rev,
                    "before": round(prev_m, 3), "after": round(cur_m, 3)}
    return None
 
if __name__ == "__main__":
    recs = load_records()
    boundary = find_regression_boundary(recs)
    print(boundary or "no regression detected")

By "bisect" I mean: rather than comparing every revision at random, order them by time and only look at the drop between adjacent revisions. Comparing mean scores in the order revisions appeared yields the from/to hashes of the moment it fell in one shot. Then git diff prompts/article_ja_draft.txt between those two wordings almost always surfaces the offending line.

Start the drop threshold generously (0.1 to 0.2). Set it too small and you'll catch the model's natural jitter as a regression, which adds noise in production.

Roll back a broken revision instantly

Once you know the offending line, reverting is immediate. Because the wording is version-controlled as a file, you just restore the previous revision.

# restore that one file to the commit matching the "from" (pre-drop) hash
git log --oneline -- prompts/article_ja_draft.txt
git checkout <good-commit> -- prompts/article_ja_draft.txt

After reverting, pin the "good" revision hash for peace of mind. To prevent recurrence, add one light guard before generation.

PINNED_GOOD = {"article_ja_draft": "9f2c1ab33de0"}
 
def assert_not_regressed(p: dict):
    pinned = PINNED_GOOD.get(p["id"])
    if pinned and p["revision"] != pinned:
        # don't silently proceed; make it noticeable (route to alerts if unattended)
        print(f"warning: {p['id']} drifted from pinned {pinned} -> {p['revision']}")

The point is to use the guard to notice, not to halt. In an unattended run, the realistic landing spot that avoids the pitfall is to keep processing while drifted but fire a notification. Halting outright drops an entire night of generation, which hurts more.

The small judgment calls that paid off

After running this for several sites over about half a year, a few design choices paid off. The essentials, as a table.

The last one mattered most. I started strict and halted on any drift, but trivial jitter ended up wasting whole runs, so leaning toward notification was more stable in practice. Stricter is not always safer.

At Dolice Labs I generate articles for Stripe memberships across several sites every day, so small differences in tone compound into the impression readers receive. That's exactly why being able to trace "which line moved it" is a quiet form of quality control that sits close to revenue.

A first step from here

Pull just your most-used prompt out into a text file outside your code, and add one prompt_revision column to your generation log. That alone turns your next investigation from "gut feeling" into "identify the window." Bisecting and rollback only become meaningful once that one column exists.

I'm still refining my own operation, but if this trims the cause-isolation work for anyone else carrying a generation pipeline alone, I'll be glad. Thank you for reading.