◈ API / SDK/2026-06-29Advanced

Guarding Gemini API Responses in CI: Snapshot and Semantic Regression Testing

Defend non-deterministic Gemini API responses with pytest snapshots plus embedding-based semantic regression detection: record/replay CI, a weekly live smoke that keeps cassettes honest, per-field drift reporting, threshold calibration, and update governance — all in working code.

Gemini API²⁴⁰ pytest³snapshot testingCI⁷regression detectionPython⁴⁷syrupyvcrpy

✦ Premium Article

Testing code that calls the Gemini API trips you up within minutes. Generative AI responses change on every run, so a plain assertion never holds:

# This will fail unpredictably
assert response.text == "Python is a general-purpose programming language."

The reflex is to mock the API — but that's a trap. A mock only verifies the mock's behavior; it stays silent when the real response structure shifts underneath you. As an indie developer running several automated pipelines for my own apps, I once had a single field quietly disappear from a model's output, and a nightly batch ran empty for days without a word. That "I never noticed" feeling is where this article starts.

Here we build on structure-preserving snapshot tests, then add a second layer that catches the regression snapshots miss — the case where the schema is identical but the content quality degrades. We'll also wire it into CI without calling the API on every run, set a threshold that separates flakiness from genuine regressions, and define a discipline for updating snapshots without rubber-stamping them.

Why Snapshots, and How They Differ From Mocks

Snapshot testing records the expected output on the first run and compares against it afterward. The point is to check structure and key fields, not exact equality.

First run  → record the response (create snapshot)
Later runs → compare against it; fail on a diff

The difference from mocking is decisive. A mock freezes the shape you imagined, so it can't notice when reality drifts from that. A snapshot freezes the shape that actually came back, so it surfaces a diff when reality moves. What you want to protect is not your assumption but the moving target that is an external API.

In Python, syrupy is the standard library.

pip install syrupy pytest pytest-recording

Build on the Current google-genai SDK

Start on the current SDK. Use google-genai (from google import genai), not the legacy google.generativeai. Create the client once and name the model explicitly, like gemini-2.5-flash.

# gemini_client.py
from google import genai
from google.genai import types
 
client = genai.Client(api_key="YOUR_GEMINI_API_KEY")
 
def classify_sentiment(text: str) -> str:
    """Thin wrapper returning sentiment JSON."""
    resp = client.models.generate_content(
        model="gemini-2.5-flash",
        contents=(
            "Analyze the sentiment of the text and respond in JSON.\n"
            "Fields: sentiment (positive/negative/neutral), "
            "confidence (0.0-1.0), explanation\n"
            f"Text: {text}"
        ),
        config=types.GenerateContentConfig(
            temperature=0,
            response_mime_type="application/json",
        ),
    )
    return resp.text

A floating alias like gemini-flash-latest is a poor test target. Aliases can be repointed to a different model behind the scenes, leaving your tests quietly validating something else. Pin a dated, versioned model name in tests and evaluate alias promotions separately with a golden set.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦A two-layer test: structure snapshots plus an embedding score that catches quality regressions the schema hides

✦Running CI without hitting the API every time, and threshold design that tells flakiness apart from real regressions

✦A snapshot-update workflow that keeps your green builds trustworthy instead of rubber-stamped

✦A weekly live-smoke job that stops stale cassettes from validating an API that no longer exists

✦Per-field drift reporting that turns a red semantic test into a diagnosis in about fifteen seconds

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

Layer One: Structure Snapshots Protect the Shape

Text content is allowed to vary, so what you verify is structure and metadata.

# test_structure.py
import json
import pytest
from gemini_client import client
from google.genai import types
 
@pytest.mark.vcr  # record/replay the API response (below)
def test_sentiment_schema(snapshot):
    """Verify the JSON output schema is preserved."""
    resp = client.models.generate_content(
        model="gemini-2.5-flash",
        contents='Sentiment of "I really enjoyed this product" as JSON. '
                 'Fields: sentiment, confidence, explanation',
        config=types.GenerateContentConfig(
            temperature=0,
            response_mime_type="application/json",
        ),
    )
    parsed = json.loads(resp.text)
 
    # Snapshot the shape, not the content
    schema = {
        "has_sentiment": "sentiment" in parsed,
        "sentiment_valid": parsed.get("sentiment") in ("positive", "negative", "neutral"),
        "confidence_is_float": isinstance(parsed.get("confidence"), float),
        "has_explanation": "explanation" in parsed,
        "finish_reason": str(resp.candidates[0].finish_reason),
    }
    assert schema == snapshot

Record first, compare afterward.

pytest test_structure.py --snapshot-update   # first: create snapshot
pytest test_structure.py                      # after: diff check

Even with temperature=0, text can shift, because internal sampling isn't perfectly deterministic. That's exactly why freezing structure rather than content is the stable choice. This layer alone reliably catches "a JSON key disappeared" or "finish_reason flipped to MAX_TOKENS."

Layer Two: Turn "Meaning Drift" Into a Number With Embeddings

Structure snapshots have a blind spot: the schema is correct but the content quality drops. The sentiment field exists and is well-typed, yet classification accuracy slipped after a model update. You protected the shape but not the meaning.

For the second layer, embed the output for a set of representative inputs and measure how far it has drifted from a baseline vector using cosine similarity.

# semantic_snapshot.py
import numpy as np
from google import genai
from google.genai import types
 
client = genai.Client(api_key="YOUR_GEMINI_API_KEY")
 
def embed(text: str) -> np.ndarray:
    r = client.models.embed_content(
        model="gemini-embedding-2",
        contents=text,
        config=types.EmbedContentConfig(output_dimensionality=256),
    )
    v = np.array(r.embeddings[0].values, dtype=np.float32)
    return v / np.linalg.norm(v)  # normalize so cosine == dot product
 
def cosine(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.dot(a, b))

In the test, store the baseline embedding under baseline/ and fail when the current output's similarity falls below a threshold.

# test_semantic.py
import numpy as np
import pytest
from pathlib import Path
from gemini_client import classify_sentiment
from semantic_snapshot import embed, cosine
 
BASELINE = Path("baseline/sentiment_review.npy")
THRESHOLD = 0.88  # determined empirically (below)
 
@pytest.mark.vcr
def test_sentiment_semantic_stability():
    current = classify_sentiment("Shipping was slow, but support was courteous.")
    cur_vec = embed(current)
 
    if not BASELINE.exists():
        BASELINE.parent.mkdir(exist_ok=True)
        np.save(BASELINE, cur_vec)
        pytest.skip("Created a new baseline; comparing from next run.")
 
    base_vec = np.load(BASELINE)
    sim = cosine(cur_vec, base_vec)
    assert sim >= THRESHOLD, f"Semantic drift: cosine={sim:.3f} < {THRESHOLD}"

With both layers, you receive "the shape broke" (layer one) and "the meaning degraded" (layer two) as separate signals. Root-causing gets faster, and you stop eyeballing full responses on every model update. On my own setup, eyeballing 40 representative inputs at roughly 30 seconds each — about 20 minutes — was replaced by a single test run (around 12 seconds via record/replay) plus a look at only the handful that failed.

Don't Hit the API Every Time — Wiring CI

Calling the real API on every test run costs money and time and runs into rate limits. With pytest-recording (vcrpy), hit the real API once to record, then replay the cassette afterward.

# pytest.ini
[pytest]
addopts = --record-mode=none

# conftest.py — keep API keys out of cassettes
import pytest
 
@pytest.fixture(scope="module")
def vcr_config():
    return {
        "filter_headers": [("x-goog-api-key", "REDACTED"),
                           ("authorization", "REDACTED")],
        "filter_query_parameters": [("key", "REDACTED")],
        "record_mode": "none",
    }

The workflow:

# When adding a new test: hit the real API exactly once to record
pytest test_semantic.py --record-mode=once
git add tests/cassettes/   # commit the cassette
# In CI from then on: replay only, no network
pytest --record-mode=none

Now CI runs offline and deterministically against the same inputs. Always mask API keys and auth headers via filter_headers before committing — skip it and the cassette keeps your key.

Threshold Design: Flakiness vs. Real Regression

The hardest part of semantic testing is the threshold. Too strict and normal variation turns the build red (flaky); too loose and real degradation slips through. Here's how I set it.

First, with the model pinned, send the same input 20 times and take the distribution of pairwise cosine similarities. The lower end of this "natural spread under identical conditions" is the noise floor you must tolerate.

# calibrate_threshold.py — measure the floor of natural variation
import numpy as np
from itertools import combinations
from gemini_client import classify_sentiment
from semantic_snapshot import embed, cosine
 
samples = [embed(classify_sentiment("Shipping was slow, but support was courteous."))
           for _ in range(20)]
sims = [cosine(a, b) for a, b in combinations(samples, 2)]
print(f"min={min(sims):.3f}  mean={np.mean(sims):.3f}  p05={np.percentile(sims,5):.3f}")

Set the threshold "a little below the p05 of the natural spread." If p05 is 0.91, put the threshold around 0.88. Then incidental wobble won't turn it red, but a shift in the distribution itself — a real regression — will. To suppress flakiness further, don't fail on a single miss; treat "2 of 3 failures" as a regression with a retry wrapper.

def stable_assert(check, attempts=3, need_fail=2):
    fails = sum(0 if check() else 1 for _ in range(attempts))
    assert fails < need_fail, f"{fails}/{attempts} failed: likely a real regression"

Before / After: Replace Brittle Tests With Structure Monitoring

Finally, rewrite a classic brittle test using everything above.

# Before: pin content with exact equality → goes red on every model update
def test_summary():
    out = summarize("...long text...")
    assert out == "This article explains Gemini testing techniques."

# After: monitor structure (layer one) and meaning (layer two) separately
@pytest.mark.vcr
def test_summary_structure(snapshot):
    out = summarize_json("...long text...")      # {"summary": str, "topics": [...]}
    parsed = json.loads(out)
    assert {
        "has_summary": isinstance(parsed.get("summary"), str),
        "topics_is_list": isinstance(parsed.get("topics"), list),
        "topics_nonempty": len(parsed.get("topics", [])) > 0,
    } == snapshot
 
@pytest.mark.vcr
def test_summary_semantic():
    out = summarize_json("...long text...")
    sim = cosine(embed(out), np.load("baseline/summary.npy"))
    assert sim >= 0.86

In the Before, a human hard-codes "the correct string," so any change turns it red and you spend effort fixing tests for reasons unrelated to the product. The After splits what you protect into shape and meaning and tells you which one broke. Only tests whose red has a clear meaning survive long-term maintenance.

A Cassette Freezes Yesterday's Reality — Add a Weekly Live Smoke

Record/replay makes CI fast and cheap, but there's a bill attached. A cassette preserves the API as it was at the moment you recorded it, which means the older your cassettes get, the more your CI is validating a Gemini API that no longer exists. A build where every offline test is green while the production batch is quietly broken is not a paradox — it's the expected outcome of a stale cassette.

This one caught me. A repository of mine held a green build on cassettes recorded six months earlier, while the actual response had changed a field's type. The replay suite never said a word. The tests didn't lie; I simply had no answer to the question of which era of reality I was testing against.

The fix is to split your suite into two jobs with different responsibilities.

Job	Trigger	Hits the API?	Scope	On failure
Offline CI	Every PR and push	No (replay)	Structure + semantic	Blocks merge
Live smoke	Weekly schedule	Yes (a few inputs)	Structure only	Files an issue, no block

Leaving the semantic layer out of the live job is deliberate. Real calls vary run to run, so a red semantic test there can't tell you whether the model shifted or the sampler simply wandered. The weekly job answers exactly one question: is the structure your cassettes encode still the structure reality returns?

Cassette freshness deserves a check of its own.

# add to conftest.py — report cassettes that have gone stale
import time
from pathlib import Path
 
CASSETTE_MAX_AGE_DAYS = 45
 
def pytest_sessionfinish(session, exitstatus):
    """Warn about aging cassettes without failing the run."""
    now = time.time()
    stale = []
    for c in Path("tests/cassettes").rglob("*.yaml"):
        age_days = (now - c.stat().st_mtime) / 86400
        if age_days > CASSETTE_MAX_AGE_DAYS:
            stale.append((c.name, int(age_days)))
    if stale:
        print("\n[cassette] stale recordings:")
        for name, days in sorted(stale, key=lambda x: -x[1]):
            print(f"  - {name}: recorded {days}d ago (limit {CASSETTE_MAX_AGE_DAYS}d)")

It warns rather than fails on purpose. Staleness isn't a defect, and making it red trains people to re-record mindlessly just to restore green. All you want here is a nudge that says it's time to go look at reality again.

The weekly job runs only the structure tests, against the live API.

# .github/workflows/live-smoke.yml
name: live-smoke
on:
  schedule:
    - cron: "0 21 * * 0"   # Mondays, 06:00 JST
  workflow_dispatch:
 
jobs:
  smoke:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements-dev.txt
      - name: Structure tests against the real API
        env:
          GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
        run: pytest test_structure.py --record-mode=none --disable-recording -q

--disable-recording sends @pytest.mark.vcr tests to the real network instead of the cassette, and — critically — doesn't overwrite anything, so your recordings stay clean. That single flag is the difference between a weekly job that verifies and one that silently re-records the very thing it was supposed to check.

Three to five representative inputs are plenty. This isn't coverage; it's a fixed observation point that tells you cheaply whether the shape moved. Spending a few tens of yen a month to avoid six months of trusting a lying cassette is a trade I take without hesitating.

The Thirty Seconds After It Turns Red: Show Which Field Drifted

When a semantic test fails with cosine=0.83 < 0.88, that line tells you almost nothing. Folding the whole payload into one vector means a degraded summary and a mangled tag array collapse into the same number. You end up reading the raw output side by side anyway, which throws away half the reason you added the second layer.

So keep a separate baseline per JSON field and print the breakdown when things fail.

# field_drift.py
import json
import numpy as np
from pathlib import Path
from semantic_snapshot import embed, cosine
 
BASE_DIR = Path("baseline/fields")
 
def _to_text(value) -> str:
    """Flatten lists and dicts into a stable string so they stay comparable."""
    if isinstance(value, str):
        return value
    return json.dumps(value, ensure_ascii=False, sort_keys=True)
 
def field_drift(payload: dict, name: str) -> dict[str, float]:
    """Per-field cosine similarity; seeds missing baselines and returns empty."""
    out: dict[str, float] = {}
    for key, value in payload.items():
        path = BASE_DIR / name / f"{key}.npy"
        vec = embed(_to_text(value))
        if not path.exists():
            path.parent.mkdir(parents=True, exist_ok=True)
            np.save(path, vec)
            continue
        out[key] = cosine(vec, np.load(path))
    return out
 
def format_report(drift: dict[str, float], threshold: float) -> str:
    rows = sorted(drift.items(), key=lambda kv: kv[1])
    lines = [f"{'field':<16}{'cosine':>8}  status"]
    for key, sim in rows:
        mark = "DRIFT" if sim < threshold else "ok"
        lines.append(f"{key:<16}{sim:>8.3f}  {mark}")
    return "\n".join(lines)

The test still gates on the worst field, but a failure now carries its own diagnosis.

# test_semantic_fields.py
import json
import pytest
from gemini_client import classify_sentiment
from field_drift import field_drift, format_report
 
THRESHOLD = 0.88
 
@pytest.mark.vcr
def test_sentiment_fields_stable():
    payload = json.loads(
        classify_sentiment("Shipping was slow, but support handled it well.")
    )
    drift = field_drift(payload, name="sentiment_review")
    if not drift:
        pytest.skip("Seeded per-field baselines; comparison starts next run.")
 
    worst_key = min(drift, key=drift.get)
    assert drift[worst_key] >= THRESHOLD, (
        f"\nworst field: {worst_key} (cosine={drift[worst_key]:.3f})\n"
        + format_report(drift, THRESHOLD)
    )

A failure reads like this:

worst field: explanation (cosine=0.804)
field             cosine  status
explanation        0.804  DRIFT
confidence         0.981  ok
sentiment          0.998  ok

sentiment and confidence are healthy; only explanation moved. That narrows the cause immediately — the classification itself is intact, so what shifted is the explanatory generation, pointing at the explanation clause of your prompt or a change in the model's output-length behavior.

Since adding the breakdown, my first-pass triage on a red semantic test went from minutes of comparing prose to about fifteen seconds of reading the failure log. The value of a test isn't that it turns red; it's that the next move is obvious the moment it does.

One caution: per-field baselines multiply the contents of baseline/fields/. Keep it to one or two representative inputs per use case. Every extra baseline is extra review surface, and review surface is exactly what erodes the governance described next.

Governance for Updating Snapshots

Left alone, snapshots and baselines drift toward "just run --snapshot-update to get green." That's the fastest path to hollow tests. Put a minimum of rules around updates.

An update is legitimate in only two cases: (1) you intentionally changed the model or prompt and thus the structure, or (2) you added or changed what you validate. When a model update silently changed the content, the answer is not to update but to investigate first. When you do update, always review the diff and keep it in its own commit.

pytest --snapshot-update
git add -p __snapshots__/ baseline/    # inspect each diff before staging
git commit -m "test: update snapshots for prompt schema change (intentional)"

Don't fold the diff into the same commit as a code change, and leave one line in the message explaining why the update is legitimate. Those two habits alone let the you of six months from now decide whether a given green build can be trusted.

Where to Start

Add exactly one layer-one structure snapshot to the single response you'd least want to break. Once it runs stably under record/replay, layer the semantic test on top for your representative inputs. Don't try to two-layer everything at once; stack one layer at a time, in order of "how much it hurts when it breaks." That, I've found, is the realistic way to keep this going.

Thank You for Reading

Gemini Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.