Don't Let Your Gemini Prompts Silently Rot — A Practical Regression Testing Playbook with Pytest

Why Pytest is the right starting point, even with dedicated tools available

There are excellent dedicated tools in this space — promptfoo, Braintrust, LangSmith — and I use them alongside Pytest depending on the project. But if you're a solo developer taking your first step, Pytest is overwhelmingly the practical choice.

Three reasons. First, your existing test infrastructure (CI, pytest -k, parallelization, coverage hooks) carries over unchanged. Second, your evaluation code is Python, the same language as your production code — the vocabulary matches, which helps a lot when you're reading failures six months from now. Third, there's no new dashboard to introduce; failures land directly in your GitHub Actions logs where you already look.

Start with Pytest, get a feel for what actually hurts, then graduate to a dedicated tool if the pain profile calls for it. Migration cost is manageable once you understand what you need.

A few numbers on cost, since solo developers care. Snapshot tests running on Flash cost a few yen per case at most. Judge tests running on Pro run a few tens of yen per case. Ten cases on a daily schedule comes out to a few thousand yen per month. I consider that cheaper than shipping one production bug I could have caught.

Minimal setup

Four dependencies are enough to start: google-genai (the new SDK), pytest, syrupy (for snapshots), and pydantic (for schema validation). Resist the urge to add more up front.

pip install google-genai pytest syrupy pydantic

Keep the project layout simple. Split it further when you pass about 30 test cases; not before.

app/
  prompts/
    review_summarizer.py
  services/
    gemini_client.py
tests/
  prompts/
    test_review_summarizer.py
    __snapshots__/

I put the test filename next to the prompt it tests, one-to-one. A small discipline, but future-me always thanks past-me for it.

Implementation 1 — Snapshot tests to defend the shape

The first thing to defend is the output's skeleton. If something that's supposed to be JSON suddenly returns prose, or drops a required field, every downstream parser falls over. Lock it down with snapshots and Pydantic.

Here's an example: a Gemini prompt that summarizes app reviews into {score, positives, concerns, one_liner}, with Pydantic pinning the schema and Pytest snapshots recording the shape.

# tests/prompts/test_review_summarizer.py
from __future__ import annotations
 
from typing import Any
 
import pytest
from pydantic import BaseModel, Field
from google import genai
from google.genai import types
 
class ReviewSummary(BaseModel):
    """Contract for a review summary. Breaking this is a loud failure."""
    score: int = Field(ge=1, le=5)
    positives: list[str] = Field(min_length=1, max_length=5)
    concerns: list[str] = Field(max_length=5)
    one_liner: str = Field(min_length=5, max_length=80)
 
SYSTEM_PROMPT = """\
You summarize a cluster of app reviews. Return:
- score: overall 1-5
- positives: 1-5 positive points
- concerns: 0-5 concerns
- one_liner: one sentence, 80 chars max
"""
 
def call_summarizer(reviews: list[str]) -> ReviewSummary:
    """Always go through the production entry point."""
    client = genai.Client()  # Reads GEMINI_API_KEY from env
    response = client.models.generate_content(
        model="gemini-2.5-flash",
        contents=[SYSTEM_PROMPT, "\n---\n".join(reviews)],
        config=types.GenerateContentConfig(
            response_mime_type="application/json",
            response_schema=ReviewSummary,
            temperature=0.2,
        ),
    )
    # `parsed` returns a Pydantic instance directly
    return response.parsed  # type: ignore[return-value]
 
@pytest.fixture
def sample_reviews() -> list[str]:
    return [
        "Too many notifications. But search is genuinely useful.",
        "Fatal: doesn't open offline. 2 stars.",
        "Snappy, love it. Would be perfect without the ads.",
    ]
 
def test_shape_is_stable(sample_reviews: list[str], snapshot: Any) -> None:
    """Make sure the schema and rough structure haven't regressed."""
    result = call_summarizer(sample_reviews)
    # Compare shape, not literal text
    shape = {
        "keys": sorted(result.model_dump().keys()),
        "positives_len_band": _band(len(result.positives)),
        "concerns_len_band": _band(len(result.concerns)),
        "one_liner_len_band": _band(len(result.one_liner), step=20),
    }
    assert shape == snapshot
 
def _band(n: int, step: int = 1) -> str:
    """Bucket into ranges so natural variation doesn't make us red."""
    lo = (n // step) * step
    return f"{lo}-{lo + step - 1}"

Three things are doing the real work here. First, the test calls through a thin wrapper (call_summarizer) rather than constructing the Gemini request inline. If you swap models later, the tests don't need touching. Second, snapshots store the shape, not the raw text. positives doesn't need to be exactly three items every run; anywhere in 1–5 is fine, so we bucket into ranges. Third, Pydantic constraints (ge, le, min_length) catch schema violations before the assertion runs. A broken schema raises; a drift in values fails the snapshot. You get two clear failure modes instead of one muddled one.

Run pytest --snapshot-update the first time to record the snapshots. From then on, any diff turns the test red. That's enough to catch "the output format changed quietly" — the most common kind of quiet regression.

Pick the snapshot granularity for the six-month-later you

A common failure mode is storing too much in the snapshot. If you dump the entire JSON, word-level drift will turn tests red every run, and eventually everyone (including you) will start reflexively running --snapshot-update — killing the entire point.

My rule of thumb: pick the granularity where, if you were reading the snapshot diff six months from now, you could still remember what you cared about. Field presence, count bands, length bands. At that level, only meaningful changes go red.

Implementation 2 — LLM-as-Judge tests for semantic quality

Shape is necessary but not sufficient. Now we evaluate whether the content is still acceptable, using Gemini 2.5 Pro as a judge.

LLM-as-Judge isn't perfect — it has biases I'll discuss later (positional, leniency). But running human evaluation on every PR isn't realistic for a solo developer, so we accept the imperfection and engineer around it in the judge prompt.

# tests/prompts/test_review_summarizer_quality.py
from __future__ import annotations
 
from typing import Literal
 
import pytest
from pydantic import BaseModel, Field
from google import genai
from google.genai import types
 
from tests.prompts.test_review_summarizer import call_summarizer
 
class Rubric(BaseModel):
    faithfulness: Literal["pass", "warn", "fail"]
    coverage: Literal["pass", "warn", "fail"]
    tone: Literal["pass", "warn", "fail"]
    reasoning: str = Field(max_length=400)
 
JUDGE_PROMPT = """\
You are evaluating a review summary. Score each axis independently.
Use pass / warn / fail. Don't be overly harsh or overly generous.
 
- faithfulness: The summary doesn't contradict the source reviews. Invented
  facts are fail.
- coverage: The summary includes the main points (positives and concerns).
- tone: Natural for an app-review summary. No overblown claims or dramatic
  phrasing.
 
In `reasoning`, write 2-4 sentences of specific observations in English.
"""
 
def judge(reviews: list[str], summary_json: str) -> Rubric:
    client = genai.Client()
    response = client.models.generate_content(
        model="gemini-2.5-pro",  # Use the stronger model as judge
        contents=[
            JUDGE_PROMPT,
            f"[Source reviews]\n{reviews}\n\n[Summary]\n{summary_json}",
        ],
        config=types.GenerateContentConfig(
            response_mime_type="application/json",
            response_schema=Rubric,
            temperature=0.0,
        ),
    )
    return response.parsed  # type: ignore[return-value]
 
@pytest.mark.parametrize(
    "reviews",
    [
        [
            "Too many notifications. But search is genuinely useful.",
            "Fatal: doesn't open offline. 2 stars.",
            "Snappy, love it. Would be perfect without the ads.",
        ],
        [
            "Won't launch after the big update.",
            "Love the new UI. The pricing tiers are confusing though.",
        ],
    ],
    ids=["mixed_feedback", "post_update_issues"],
)
def test_summary_quality(reviews: list[str]) -> None:
    summary = call_summarizer(reviews)
    r = judge(reviews, summary.model_dump_json())
 
    # fail => red. Tolerate one warn (realistic operating point).
    failures = [k for k, v in r.model_dump().items() if v == "fail"]
    warns = [k for k, v in r.model_dump().items() if v == "warn"]
 
    assert not failures, f"quality fail: {failures} / reasoning: {r.reasoning}"
    assert len(warns) <= 1, f"too many warns: {warns} / reasoning: {r.reasoning}"

The key design choice is splitting the evaluation into three independent axes. If you ask the judge to "evaluate faithfulness, coverage, and tone together," it tends to collapse into a vague overall impression and miss individual red flags. Scoring each axis separately with a short reasoning string makes it obvious, on failure, which axis failed and why.

The second important choice is tolerating one warn. LLM-as-Judge doesn't give you reliable 0/1 verdicts, and "all axes must always be pass" is an aspiration, not an operating point. In practice, "fail is red immediately; one warn tolerated" has landed as the sweet spot between noise and missed regressions.

Making the judge trustworthy

When the judge gets lenient, it's usually because the rubric is vague or the examples are missing. Adding one concrete example each for pass / warn / fail in the rubric tightens judgments noticeably. I also put a Pydantic schema on the judge so it has to produce structured output; free-form judgments drift in length and stability even at temperature zero.

Model selection for the judge matters too. If generation runs on Flash, run the judge on Pro — a cross-model setup reduces self-reinforcement bias (the tendency to go easy on output from the same model). Cost goes up, but judge calls are one per test case, so the dollar impact stays in a reasonable range.

Implementation 3 — Pairwise comparison to direct prompt improvements

Snapshots and single-axis judging protect you from regression. But when you're deciding whether a new prompt is actually better than the old one, pairwise comparison earns its keep.

# tests/prompts/test_pairwise.py
from __future__ import annotations
 
from typing import Literal
 
from pydantic import BaseModel, Field
from google import genai
from google.genai import types
 
class PairwiseVerdict(BaseModel):
    winner: Literal["A", "B", "tie"]
    reason: str = Field(max_length=300)
 
PAIRWISE_PROMPT = """\
Compare summaries A and B on faithfulness, coverage, and tone.
Return A / B / tie as the overall winner, with a short reason.
Before deciding, mentally swap A and B to guard against positional bias.
"""
 
def pairwise(reviews: list[str], summary_a: str, summary_b: str) -> PairwiseVerdict:
    """Query the judge twice with orders swapped to reduce positional bias."""
    client = genai.Client()
 
    def _ask(first: str, second: str) -> PairwiseVerdict:
        resp = client.models.generate_content(
            model="gemini-2.5-pro",
            contents=[
                PAIRWISE_PROMPT,
                f"[Source reviews]\n{reviews}\n\n[A]\n{first}\n\n[B]\n{second}",
            ],
            config=types.GenerateContentConfig(
                response_mime_type="application/json",
                response_schema=PairwiseVerdict,
                temperature=0.0,
            ),
        )
        return resp.parsed  # type: ignore[return-value]
 
    v1 = _ask(summary_a, summary_b)
    v2 = _ask(summary_b, summary_a)
    # In v2 the labels are flipped
    v2_flipped = PairwiseVerdict(
        winner={"A": "B", "B": "A", "tie": "tie"}[v2.winner],
        reason=v2.reason,
    )
    if v1.winner == v2_flipped.winner:
        return v1
    return PairwiseVerdict(
        winner="tie",
        reason=f"Verdict flipped on order swap: {v1.reason} / {v2_flipped.reason}",
    )

Positional bias is the judge's tendency to favor whichever option was shown first. I started with a single call per comparison, and discovered that swapping A and B flipped the verdict in about 20% of cases. Only accepting a verdict when both orders agree cuts most of that noise cheaply.

When you change a prompt, running pairwise between the old and new versions on your test cases gives you a proper win-rate number. It's a fast way to short-circuit the "is this change an improvement or just my taste?" spiral that otherwise eats afternoons.

Wire it into CI and let it block you when it fails

Once the tests exist, they need to run on every change. I use GitHub Actions to run prompt tests as a separate job on each pull request.

# .github/workflows/prompt-tests.yml
name: prompt-tests
 
on:
  pull_request:
    paths:
      - "app/prompts/**"
      - "tests/prompts/**"
  schedule:
    # Weekly heartbeat. Catches silent shifts from model updates.
    - cron: "0 2 * * 1"
 
jobs:
  pytest:
    runs-on: ubuntu-latest
    concurrency:
      group: prompt-tests-${{ github.ref }}
      cancel-in-progress: true
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements-dev.txt
      - name: Run prompt tests
        env:
          GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
        run: pytest tests/prompts -q --snapshot-warn-unused

Three details matter. The paths filter limits runs to prompt or test changes — running on every push inflates cost, which creates the incentive to skip the tests. The concurrency block cancels superseded runs on the same branch; free-tier quota vanishes surprisingly fast without this. And the weekly schedule gives you a heartbeat independent of your own changes — model updates shift behavior during weeks when no one touched anything, and if Monday morning's run is red, you know it wasn't you.

Judge-backed tests get expensive, so I mark them with @pytest.mark.judge and skip them on PRs with -m "not judge", running the full judge suite only on the nightly schedule. Snapshots every PR, judges nightly, pairwise on prompt edits — that three-tier rhythm matches solo-developer cost and noise realities well.

What to do when the tests go red

When a prompt test fails, rushing to patch it tends to break things further. Have a fixed procedure ready.

First, separate "snapshot failure" from "judge failure." The former means the shape of the output changed; the latter means the content changed. Don't try to diagnose on instinct.

For shape changes, read the diff carefully. If keys appeared or disappeared, the model's schema adherence weakened — add an example to the prompt or paste the schema explicitly. If a length band shifted, rewrite the length constraint. If it's genuine natural variation, updating the snapshot is a legitimate call.

For content changes, read the judge's reasoning. Distinguish "it drifted toward better" from "it got worse." A common pattern: a prompt tweak improves one axis while weakening another that used to be strong. When that happens, rather than reverting, the right move is usually to find wording that keeps both axes strong.

A personal rule for dealing with red

LLM tests are noisier than unit tests. Accept that and decide ahead of time not to emergency-respond to a single red. My personal rule: one red triggers a rerun tomorrow; two reds in a row trigger investigation; three reds trigger a revert. Deciding the escalation policy up front keeps your emotional state out of the loop. It also makes conversations easier when you eventually extend this to a team.

Pitfalls that cost me real time

After running this setup for a few months, these are the traps I actually fell into. Writing them down so three-months-from-now me (and you) won't lose a day to them.

One: don't over-trust seed determinism. Gemini API accepts a seed parameter, but behavior under the same seed drifts when the model version updates underneath you. Assuming "today's red should match yesterday's green because the seed is the same" burns hours. Seeds help with within-session reproducibility, not cross-week comparison.

Two: don't let the judge grade itself. Using the same model (or the same prompt template) for generation and judging produces self-reinforcement bias — the judge gets soft. Cross the models (Flash generates, Pro judges) or at minimum write the judge prompt with a different vocabulary. If your judge prompt reuses phrasing from the generation prompt, the judge starts echoing the generator's framing.

Three: don't let your test dataset calcify. Testing on the same inputs month after month makes the prompt overfit to them. I spend 30 minutes at the end of each month pulling 2–3 new cases from production logs into parametrize. Skip this, and you land in the worst case: tests green, production degrading. When importing from logs, mask emails, phone numbers, and personal names before the cases ever touch the repo.

Four: don't ignore cost and latency in CI. Snapshots and judges both green doesn't mean healthy if tokens-per-request tripled. Log response.usage_metadata and assert on it the same way you assert on output shape. The cost mindset in Gemini API Cost Optimization Guide is the right companion read.

Five: keep failure alerts to one channel. PR comments plus Slack plus email means you'll eventually watch none of them. I route to a single Slack DM, and respect the "red CI is never ignored" rule religiously. Response times collapsed once I made that change.

A few more I picked up later

Six: regression-test the judge prompt itself. It's easy to focus on production prompts and forget that a tweak to the judge prompt can flip previously passing cases overnight. When I edit the judge, I rerun the full suite on committed prompt outputs to confirm the verdict set is identical before shipping. I automate this with a small judge_self_check.py.

Seven: don't mix English and Japanese in one test. Gemini is multilingual, but the judge's calibration subtly differs depending on whether it processes the rubric vocabulary in English or Japanese. Keep Japanese tasks with a Japanese judge prompt and English tasks with an English one. Judgment consistency improves noticeably.

Eight: log failing runs' reasoning to disk. pytest -v output scrolls off. I hook conftest.py on failure to append the judge's reasoning to a JSON file, and read it once a month. Seeing "which axis fails most" makes the priority of the next prompt edit obvious.

Growing the test dataset as a deliberate practice

After a while you realize the quality of your test dataset determines the ceiling of your whole testing setup. A handful of cases from development are enough to start, but months of production teach you that real user inputs are more varied and more extreme than you imagined.

Harvest the hard cases from production logs

My monthly routine pulls three kinds of cases from production logs. Cases the judge marked warn: not quite failures, but border examples worth catching next time they appear. Cases with user feedback: if a user told you a summary was "off," that exact input belongs in regression. Cases where latency or tokens exceeded 2× the median: output quality might be fine, but the cost profile is worth protecting against.

Before anything touches the repo, mask emails, phone numbers, card numbers, and named entities with Pydantic validators or a compact regex pass. Without that, future-you can't revisit the logs safely, and the repo starts accumulating risk it shouldn't hold.

Name the cases after their point

Using case_01, case_02 as test IDs is a trap. When a test fails six months later, you won't remember what it was protecting. I write short English tags into ids: mixed_feedback, post_update_issues, very_short_input, long_review_with_emoji. Tiny change, huge effect on how quickly future-me understands the failure.

Think of the test code as a note to your future self. Treat it that way and long-term maintenance gets dramatically less painful.

Quarterly dataset health check

Every three months, I walk the full case list and ask: are too many cases now redundant? Are any cases pinning us to an old model's quirks? After a few prompt improvements on the same input, you can end up with a prompt overfit to one case. Pruning stale cases is as much a part of "growing" the dataset as adding new ones.

Living with flaky judges without giving up the signal

One of the harder things to accept about LLM-as-Judge is that a small fraction of runs will flip their verdict for no identifiable reason. This is uncomfortable for anyone coming from deterministic testing, because the natural instinct is to hunt down the flake and eliminate it. With judges, the flake is often a property of the tool itself, not a bug in your code.

I've settled on a few rules that make this tolerable. First, never gate merges on a single judge call. Use best-of-three with majority voting at the PR-gate level, accept a small cost increase, and stop second-guessing verdicts in between. Second, separate "judge failed to produce a valid JSON" from "judge produced a verdict of pass/fail". The former is retryable with exponential backoff up to three attempts; the latter is a real signal and should not be retried until it says what you wanted. Mixing these two failure modes is how test suites turn into slot machines.

Third, keep a small bucket of cases marked as "judge-known-noisy". When a case flips verdicts on stable prompt code more than twice in a month, it goes into that bucket and is reviewed by hand. Sometimes the rubric is ambiguous and needs sharpening; sometimes the expected output is genuinely borderline and a human should just own the decision. Either way, flagging these cases keeps them from polluting your signal-to-noise ratio on the rest of the suite.

The mental model I've found useful is to treat the judge as a noisy sensor, not a reference implementation. A noisy sensor is still informative; you just have to read it carefully and combine it with redundancy, not trust any single reading. Once you internalize that, the anxiety of "but what if the judge is wrong" fades. The correct answer is usually "it sometimes is, and the system is designed to absorb that". That's the posture that lets you actually ship.

A closer look at what actually saves time

Running this setup across several side projects and a couple of production services, the surprise was how much time the two-layer split saves outside of test failures themselves. A few examples worth calling out.

Code review conversations get shorter

When you open a pull request that edits a prompt, reviewers used to ask "but did you test this?" and you'd handwave something about trying a few inputs in a notebook. Now the PR arrives with the snapshot diff and a judge summary attached. The review conversation shifts from "did you verify this" to "is the new snapshot actually what we want," which is a much more productive question to argue about.

If you're working alone, this matters less on day one, but it matters enormously the moment a second person joins the project. Prompt-handover conversations become "read the test cases" instead of "let me tell you about all the weird inputs I've seen," which is a much better way to transfer knowledge.

You stop second-guessing yourself after shipping

Before I had this setup, every prompt edit came with 24 hours of low-grade anxiety — watching the app, reading user reports, wondering if I'd quietly introduced drift. The snapshot + judge combination replaces that anxiety with something measurable. Either the tests are green and you can stop watching, or they're red and you know exactly what to fix. Both outcomes are strictly better than the "is it fine? I think it's fine?" state.

Regression becomes a concrete, shareable artifact

When a prompt change does hurt quality, having the failure expressed as a concrete judge reasoning string is much easier to share than "trust me, the summaries feel worse now." You can point at the exact observation and the exact axis that failed. That single fact changes how you talk about prompt quality with teammates, stakeholders, or even your future self reading Git history.

Thinking about which prompts to test first

Not every prompt deserves the same treatment. Over time I've settled into a rough prioritization based on two axes: how visible the output is to users, and how often the prompt changes.

Prompts that are high-visibility and high-change (core user-facing features that you iterate on) are the obvious first candidates. Everything I've described belongs here. Medium-visibility prompts (admin dashboards, internal tools that still affect behavior) deserve at least snapshot tests — the shape protection alone is worth it. Low-visibility, low-change prompts (one-off scripts, throwaway experiments) probably don't need tests at all; the cost of writing them outweighs the expected benefit.

The trap is treating all prompts as if they need the full two-layer setup. That turns prompt testing into busywork, tests get skipped, and the whole practice loses credibility. Start with the prompts that will actually cause pain if they break, and let the coverage grow from there.

Snapshot-first, judge-when-you-need-it

A specific sequence I've found useful: add snapshot tests to a new prompt immediately, but don't add judge tests until the prompt has been running in production for a week or two. Two benefits. First, you avoid baking in premature assumptions about what "good output" looks like — the shape tends to stabilize faster than the quality rubric. Second, you give yourself time to see what failure modes actually appear in production, which tells you what the judge rubric should actually measure.

I used to add both layers simultaneously and found myself rewriting the judge rubric three times in the first month, every time I learned something new about how users were actually using the feature. Now I wait.

What to measure once the tests are in place

Once the testing loop is running, the next question is what metadata to accumulate over time. A few small things make the data useful months later.

Log the model version string on every judge call. When Google ships a Gemini update, you want to be able to split "pre-update quality" from "post-update quality" in retrospect. This is about two lines of code and saves you from guessing six months from now.

Log token counts alongside quality verdicts. Plotting quality score against tokens-per-call tells you whether prompt improvements came at a cost, and whether cost optimizations quietly hurt quality. Without this data, you're debating on vibes.

Log the case ID with every failure. If mixed_feedback fails three times in two months, that case either has an unstable prompt behind it or is too strict a rubric — either way, it deserves attention. Without the ID, you can't tell whether failures are clustered or spread.

None of this requires a dedicated observability stack. A single CSV file written from conftest.py is enough for the first several months. The point is not to build a dashboard; it's to make sure you can answer "did this get worse?" with a grep instead of a feeling.

Where to go from here

Just one thing tonight is enough. Pick the scariest single prompt in your production code, and write snapshot tests for it. Judges can wait. Even shape-level defense catches roughly 70% of "it broke without anyone noticing" failures.

Once you start treating prompts with the same care as code, your iteration speed visibly improves. The tests that look like a detour turn out to be the shortcut. From here, you grow the setup to fit your project, case by case, month by month.