An agreement-rate gate approves the model swap, then your category shares move

Feeding identical input to both models cuts the requirement fivefold

If input variation is the problem, hold the input fixed. Send the same image to both models, and for each image subtract "was this the old label" from "is this the new label." The value is -1, 0, or +1, and the 92% that agree all contribute exactly 0.

Only the disagreeing images generate variance, and the input skew cancels in the subtraction. Measuring power again against the same drift:

Catching the top-category shift with 75% probability takes 6,000 items by production observation and 1,200 by paired comparison — roughly a fifth. Even the third category at 7.8% reaches 64.9% power at 2,500 items.

At 300 images a day, the paired check needs four days' worth of images. What had been three weeks of watching after promotion became four days before it.

Implementing the promotion check

Here is the shape I actually use, with dependencies stripped back. The classification call is the same one the normal batch makes; the only difference is sending each image twice and lining the answers up.

import os, json, math
from collections import Counter
from google import genai
from google.genai import types
 
CATEGORIES = ["landscape", "night-sky", "flower", "animal", "abstract"]  # 30 in practice
OLD_MODEL = "gemini-3.5-flash"
NEW_MODEL = "gemini-3.7-flash"
 
client = genai.Client(api_key=os.environ.get("GEMINI_API_KEY", "YOUR_API_KEY"))
 
SCHEMA = types.Schema(
    type=types.Type.OBJECT,
    required=["category"],
    properties={"category": types.Schema(type=types.Type.STRING, enum=CATEGORIES)},
)
 
def classify(model: str, image_bytes: bytes) -> str | None:
    """One image, one category. Failures return None and the image forms no pair."""
    try:
        res = client.models.generate_content(
            model=model,
            contents=[
                types.Part.from_bytes(data=image_bytes, mime_type="image/jpeg"),
                "Choose exactly one category that best fits this wallpaper.",
            ],
            config=types.GenerateContentConfig(
                response_mime_type="application/json",
                response_schema=SCHEMA,
            ),
        )
        return json.loads(res.text)["category"]
    except Exception as exc:  # a half-successful pair is unusable, so drop it
        print(f"[warn] {model}: {type(exc).__name__}: {exc}")
        return None

And the decision side — a two-sided test on the mean paired difference, run per category.

Z_975 = 1.959964  # two-sided 5%
 
def paired_report(pairs: list[tuple[str, str]]) -> list[dict]:
    """pairs holds (old_label, new_label). Drop unpaired images before calling."""
    n = len(pairs)
    if n < 200:
        raise ValueError(f"{n} pairs is not enough to decide (200 minimum)")
 
    old_share = Counter(o for o, _ in pairs)
    rows = []
    for cat in CATEGORIES:
        d = [(1 if nw == cat else 0) - (1 if od == cat else 0) for od, nw in pairs]
        mean = sum(d) / n
        var = sum((x - mean) ** 2 for x in d) / (n - 1)
        se = math.sqrt(var / n) if var > 0 else 0.0
        base = old_share[cat] / n
        rows.append({
            "category": cat,
            "old_share": base,
            "delta": mean,                       # absolute change in share
            "relative": mean / base if base else 0.0,
            "significant": se > 0 and abs(mean) / se > Z_975,
            "ci_half_width": Z_975 * se,
        })
    return rows

The var > 0 branch turned out to be necessary, and it was the first trap I hit. In small categories neither model may pick the label even once, every difference is 0, and the standard error collapses to zero, taking the division with it as a ZeroDivisionError. The workaround is to pass the row with significant set to False instead of raising — the right reading, since no difference was observed.

One more thing surfaced only in production: how to treat images that failed to form a pair. Counting an image where only one model returned a 500 as "no change" dilutes the difference on days with more failures. Unpaired images are now excluded from the decision, and if the exclusion rate passes 5% the whole decision is deferred to the next day.

Two thresholds, one relative and one absolute

Stopping on significance alone means that as sample size grows you start catching shifts too small to matter. In practice I use two layers on top of it:

1. The difference is significant (significant from the harness above)
2. The relative change exceeds 5% (abs(relative) > 0.05)
3. That change amounts to at least 30 images per day (abs(delta) * daily_volume >= 30)

The third condition exists because an 8% relative move in a category holding 1.1% of the catalog shifts only a handful of images a day, and nothing visible changes. Back when the relative threshold stood alone, noise in small categories turned the gate red regularly, and each time I looked at it by hand and concluded it was fine. When the judgment is the same every time, it belongs in the condition.

Where to draw the stop line is a product question, not a statistical one. Not delegating that part to the test is, I think, why this version survived contact with the actual schedule.

If you are processing over ten thousand images a day, production observation reaches usable power within days, and building the paired harness may not repay the effort. At a few hundred to a few thousand images, with the promotion call resting on one person, I would run the paired comparison the day before the switch. Reversing a decision costs far less before promotion than after it.

What I removed when this went into production

I kept monitoring the production output distribution in parallel at first, then dropped it. Once the power figures were in front of me, a green lamp there carried no information, and a monitor that cannot fail is more dangerous than no monitor at all.

What stayed is a small step that writes the 1,200 pairs used for the decision, the timestamp, and both model IDs into a single file. It lets me reconstruct which way I went on a given day. The same check runs every time a model generation moves, so keeping the image set reusable gives successive decisions a common basis for comparison.

For the migration process as a whole, running shadow traffic to move safely between Gemini models covers the surrounding design; this article zooms into the decision step alone. The operational side of running two models over wallpaper classification is in three weeks of running Gemini 3 Pro and 2.5 Pro side by side on wallpaper categories.

Where to start

If an agreement rate is currently deciding your model swaps, send 500 images through both models at the next changeover and print the per-category share difference. Even at 92% agreement, a 7% move in the top categories will be visible right there.

I spent a long time watching whether the labels were correct, and missed that the distribution of correct labels had changed. If this saves someone the same detour, that is enough.