Green Tests, Dead Production — How Recorded Fixtures Hide a Model Retirement, and a Freshness Gate to Catch It

The surprise: cassettes are your most accurate ledger

Before this exercise, I treated cassettes as noise in a dependency audit. When you grep a codebase for model IDs, hits inside test fixtures feel like something to filter out.

The opposite turned out to be true.

Model references in application code hide behind indirection — environment variables, config files, stable aliases. A string scan over source code loses track at every one of those layers. But the modelVersion recorded in a cassette is the resolved model name at the moment the API actually answered. However many layers of indirection your code has, the recording shows the model you really called.

A recorded fixture is simultaneously the device that hides a retirement and the most truthful record of the dependency. Once I saw that dual nature, the plan changed from "mock less" to "read the cassettes as a ledger." The audit procedure itself is the one I described in Before the August 17 image model shutdown, find out where you call it from — the only change is adding your cassette directory to the scan targets, and the quality of the audit improves noticeably.

Implementing the freshness gate — quarantine cassettes before pytest runs

If the cassettes are a ledger, reading them should be automated. This gate scans every cassette at pytest session start and fails on recordings of soon-to-retire models, or recordings that are simply too old.

# tests/freshness_gate.py
"""Cassette freshness gate: flag retiring models and stale recordings."""
import datetime as dt
import json
import pathlib
 
# Source: ai.google.dev/gemini-api/docs/deprecations (checked 2026-08-05)
DENYLIST = {
    "imagen-4.0-generate-001":        dt.date(2026, 8, 17),
    "imagen-4.0-ultra-generate-001":  dt.date(2026, 8, 17),
    "imagen-4.0-fast-generate-001":   dt.date(2026, 8, 17),
    "gemini-robotics-er-1.6-preview": dt.date(2026, 8, 31),
}
WARN_DAYS = 21     # start failing 21 days before retirement
MAX_AGE_DAYS = 90  # recordings older than 90 days demand re-recording
 
def scan(cassette_dir: pathlib.Path, today: dt.date | None = None) -> list[str]:
    today = today or dt.date.today()
    problems = []
    for path in sorted(cassette_dir.glob("*.json")):
        data = json.loads(path.read_text())
        # prefer the resolved model name; fall back to the requested one
        model = data.get("response", {}).get("modelVersion") \
            or data.get("request", {}).get("model", "")
        retire = DENYLIST.get(model)
        if retire and (retire - today).days <= WARN_DAYS:
            problems.append(
                f"{path.name}: recorded model '{model}' retires {retire.isoformat()} "
                f"({(retire - today).days} days left)")
        recorded = data.get("recorded_at")
        if recorded:
            age = (today - dt.datetime.fromisoformat(recorded).date()).days
            if age > MAX_AGE_DAYS:
                problems.append(f"{path.name}: recording is {age} days old (> {MAX_AGE_DAYS})")
    return problems

Wiring it into conftest takes a few lines, with an explicit escape hatch for emergencies.

# tests/conftest.py (addition)
def pytest_sessionstart(session):
    import os
    if os.environ.get("SKIP_FRESHNESS_GATE") == "1":
        return
    from tests.freshness_gate import scan
    problems = scan(CASSETTE_DIR)
    if problems:
        import pytest as _pytest
        _pytest.exit("cassette freshness gate failed:\n  " + "\n  ".join(problems), returncode=2)

The suite that was all green a moment ago now says this:

! _pytest.outcomes.Exit: cassette freshness gate failed:
  imagen_fast_batch.json: recorded model 'imagen-4.0-fast-generate-001'
      retires 2026-08-17 (12 days left)
  imagen_fast_filtered.json: recorded model 'imagen-4.0-fast-generate-001'
      retires 2026-08-17 (12 days left)
  imagen_fast_portrait.json: recorded model 'imagen-4.0-fast-generate-001'
      retires 2026-08-17 (12 days left)

"12 days left," delivered as a test failure. That is exactly the behavior I wanted: instead of tracking retirement dates on a calendar, the evidence of the dependency — the recording — speaks for itself.

Measured overhead, and how to pick the thresholds

A slow gate gets deleted. I generated 200 cassettes (~2KB each) and timed the scan 50 times on the VM (Python 3.10.12).

That is fully buried inside pytest's own startup cost. Performance is not a reason to skip this.

The thresholds deserve more thought than the code. Note the 101 stale flags above: turning MAX_AGE_DAYS on from day one will bury an established suite in failures. My recommended order:

1. Start with DENYLIST checks only. Those failures are work you genuinely must do, and the failure list doubles as your migration checklist.
2. Set WARN_DAYS to 21. Migration plus re-recording needs two to three weeks of slack; learning about a retirement on retirement day is not learning, it is triage.
3. Keep MAX_AGE_DAYS as a warning at first, and promote it to a failure only after re-recording has become routine.

Resisting the urge to enforce everything at once matters. The point of the gate is to build a habit of owning recording freshness — not to hand your team 101 red tests one morning.

Fitting it into daily operation, and the hole that remains

CI integration is essentially done the moment the gate lives in conftest: every ordinary test run doubles as quarantine. When an emergency deploy truly must go through, SKIP_FRESHNESS_GATE=1 exists — and using it should itself be logged.

One honest limitation: DENYLIST is maintained by hand. If you stop reading the changelog, the gate quietly rots. I piggyback the update on my daily reference-data routine, and I deliberately skipped auto-scraping the deprecations page because its structure is not stable enough to trust. The gate reliably communicates the retirements you know about; it does not discover the ones you do not. Keep that boundary in mind.

This week, run scan() once against your own cassette directory. The failure list it prints is your migration to-do list for August 17. Trust your green tests — but interrogate, just once, what the green is standing on. That single interrogation is what makes retirement day a quiet one.

I hope this saves you a scramble. I write this as someone whose own gate said "12 days left" and finally got me moving.