Where to Adopt Gemini 3.5 Flash GA First — Per-Workload Evaluation and a Staged Rollout with a Model Router

Writing the Evaluation Harness — Measure on Your Real Tasks

Published benchmarks are not a proxy for your tasks. The harness below pulls representative cases out of the production pipeline and feeds identical inputs to old and new models. The problem it solves: getting the numbers a migration decision actually needs — latency, tokens, and format pass rate — from a single command.

# compare_models.py — measure old vs. new models on real production tasks
import json
import os
import statistics
import time
 
from google import genai
 
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
 
MODELS = ["gemini-2.5-flash", "gemini-3.5-flash"]
 
# Representative cases extracted from the production pipeline.
# Each case: {"prompt": "...", "required_keys": ["title", "tags"]}
with open("eval_cases.json", encoding="utf-8") as f:
    CASES = json.load(f)
 
 
def run_case(model: str, case: dict) -> dict:
    started = time.perf_counter()
    response = client.models.generate_content(
        model=model,
        contents=case["prompt"],
        config={
            "response_mime_type": "application/json",
            "temperature": 0.2,
        },
    )
    elapsed = time.perf_counter() - started
    usage = response.usage_metadata
    return {
        "latency": elapsed,
        "input_tokens": usage.prompt_token_count,
        "output_tokens": usage.candidates_token_count,
        "text": response.text,
    }
 
 
def is_valid(case: dict, text: str) -> bool:
    # Use the downstream parser's actual requirements as the pass condition
    try:
        data = json.loads(text)
    except json.JSONDecodeError:
        return False
    return all(key in data for key in case["required_keys"])
 
 
for model in MODELS:
    latencies = []
    valid = 0
    in_tokens = 0
    out_tokens = 0
    for case in CASES:
        result = run_case(model, case)
        latencies.append(result["latency"])
        in_tokens += result["input_tokens"]
        out_tokens += result["output_tokens"]
        if is_valid(case, result["text"]):
            valid += 1
    print(
        f"{model}: p50={statistics.median(latencies):.2f}s "
        f"valid={valid}/{len(CASES)} "
        f"tokens(in/out)={in_tokens}/{out_tokens}"
    )

The key design choice is that is_valid encodes the downstream parser's exact requirements, not a generic quality score. Measure against the conditions under which your system actually breaks; abstract that away and the measurement loses its meaning.

A run looks like this (60-case example):

gemini-2.5-flash: p50=1.82s valid=57/60 tokens(in/out)=48210/15890
gemini-3.5-flash: p50=1.41s valid=58/60 tokens(in/out)=48210/17820

Reading the Results — Check Format Stability Before Speed

Here is what my measurements (60 classification cases, 40 metadata cases) showed, with interpretation. The values depend on time of day and prompts, so read them as trends rather than absolutes.

Latency improved consistently, around 20 percent at p50. The headline "four times faster" claim applies to benchmark conditions; with forced JSON output on real tasks, the gain settles around this level. Even so, the total runtime of my nightly batches shrank noticeably.

The surprise was output token growth. Given the same instructions, 3.5 Flash is slightly more verbose — my output tokens rose a bit over 10 percent. At equal unit pricing, a faster model can still nudge your bill upward. For length-capped jobs like description generation, I tightened max_output_tokens and the character limits in the prompt to offset it.

Format pass rates were roughly equal between the two, but the failures differed in kind. When 2.5 Flash failed, it tended to drop required keys; when 3.5 Flash failed, it tended to add extra keys. If your downstream code ignores unknown keys, no harm done — but if you validate against a strict schema, be aware that the failure mode shifts with the migration.

The one workload with a clear accuracy win was agentic multi-step tasks: pass rates went from the 70–80 percent range to around 90 percent in my testing. Less hesitation in tool selection meant fewer retries. This is exactly the territory where 3.5 Flash earns its billing.

Broken out per workload, "we moved to 3.5 Flash" turns out to mean four quite different things:

Notice that the multi-step row shows no per-call latency win yet still lands on "migrate." A single call is no faster; what changed is that fewer retries were needed, so time-to-completion and total tokens both dropped. Measure only at the call level and that improvement never shows up in your table.

Fix Your GO / Canary / Hold Thresholds Before You Measure

Numbers alone do not settle anything if you have not decided in advance what counts as good enough. A latency win is something you feel immediately; a 12 percent bump in output tokens is something you notice on next month's invoice.

So before judging the four pipelines, I wrote down the thresholds first:

The exact cutoffs matter less than the fact that they existed before the data did. Thresholds chosen afterward have a way of landing just on the friendly side of your results.

Encoding them keeps the same yardstick available as workloads get added:

# decide.py — turn a measurement summary into an adoption call
from dataclasses import dataclass
 
 
@dataclass
class Measured:
    workload: str
    valid_rate_old: float   # 0.0 - 1.0
    valid_rate_new: float
    p50_old: float          # seconds
    p50_new: float
    out_tokens_old: int
    out_tokens_new: int
    voice_visible: bool     # is the model's "voice" visible to end users?
 
 
def decide(m: Measured) -> str:
    # Voice-visible workloads never auto-GO, however good the numbers look
    if m.voice_visible:
        return "HOLD"
 
    valid_delta = (m.valid_rate_new - m.valid_rate_old) * 100        # points
    latency_delta = (m.p50_new - m.p50_old) / m.p50_old              # ratio
    token_delta = (m.out_tokens_new - m.out_tokens_old) / m.out_tokens_old
 
    if valid_delta < -2 or latency_delta > 0.10 or token_delta > 0.30:
        return "HOLD"
    if valid_delta < -0.5 or latency_delta > 0 or token_delta > 0.15:
        return "CANARY"
    return "GO"
 
 
if __name__ == "__main__":
    batch = Measured(
        workload="categorize",
        valid_rate_old=57 / 60,
        valid_rate_new=58 / 60,
        p50_old=1.82,
        p50_new=1.41,
        out_tokens_old=15890,
        out_tokens_new=17820,
        voice_visible=False,
    )
    print(batch.workload, decide(batch))  # -> categorize GO

Short-circuiting on voice_visible is a deliberate asymmetry. Tone consistency cannot be certified by a pass-rate number, so I made sure no numeric result can override it. Run the review-reply workload through this function and it returns HOLD no matter what else improved.

One more deliberate choice: the canary condition on latency is "greater than zero," not "up to +10%." If a newer model makes something slower at all, it gets partial traffic for a while, even when the regression sits inside tolerance.

Staged Rollout with a Model Router — Reversible in One Line

Once the measurements settle the policy, don't rewrite model names scattered through your code. Put a single router in front that maps workload names to model IDs. The problem this solves: switching and reverting per workload, instantly, without a deploy.

# model_router.py — a model router with instant env-based rollback
import os
import random
 
DEFAULT_ROUTES = {
    "categorize": "gemini-3.5-flash",    # classification batch: migrated
    "metadata": "gemini-3.5-flash",      # metadata generation: 50% canary
    "review_reply": "gemini-2.5-flash",  # review replies: deliberately held back
    "agentic": "gemini-3.5-flash",       # multi-step tasks: migrated
}
 
# Fraction of traffic routed to the new model (unspecified = 100%)
CANARY_RATIO = {
    "metadata": 0.5,
}
 
FALLBACK_MODEL = "gemini-2.5-flash"
 
 
def resolve_model(workload: str) -> str:
    # MODEL_OVERRIDE_<WORKLOAD> wins over everything (emergency rollback)
    override = os.environ.get(f"MODEL_OVERRIDE_{workload.upper()}")
    if override:
        return override
    model = DEFAULT_ROUTES.get(workload, FALLBACK_MODEL)
    ratio = CANARY_RATIO.get(workload)
    if ratio is not None and random.random() >= ratio:
        return FALLBACK_MODEL
    return model

With this in place, incident response is just adding MODEL_OVERRIDE_METADATA=gemini-2.5-flash to the environment and restarting. No code changes, which means you can revert from a phone at midnight. I only use canary ratios on workloads where format violations are detectable downstream, like metadata generation; anything without a detection mechanism gets switched all-or-nothing. Half-mixed traffic makes bug reports impossible to reproduce. For completeness, here is the order I actually followed. From landing the router to putting the first pipeline on the new model, it went in five stages:

1. Deploy the router alone. Every workload still points at the old model, so behavior does not change at all. The point is to have the rollback lever in place before you need it
2. In staging, set MODEL_OVERRIDE_CATEGORIZE to the new model and run the same 60 cases at production-equivalent concurrency
3. Change DEFAULT_ROUTES in production for exactly one workload. I'd recommend never moving two at once — you lose the ability to attribute a failure
4. Wait 24 hours, then compare error rate, p50, and output token volume against the same weekday on the old model. Matching the weekday matters because my batch volume shifts on weekends
5. If it holds, move to the next workload. If anything looks off, add one environment variable to revert and do not advance until you know why

Running those five stages across four pipelines took nine days end to end. Compressing it into a single day is not possible without giving up the 24-hour observation window — the waiting is longer than the work.

If you want continuous output-diff observation on live traffic, you can attach shadow execution behind the router. I covered that pattern in Shadow traffic for safe Gemini model migrations — measuring output diffs in production, so I'll skip it here.

Watch for Unexpected Keys — the Monitoring That Pays Off Weeks Later

The habit I measured — 3.5 Flash failing by adding keys rather than dropping them — bites weeks after the switch rather than on day one. As long as the required keys are present, the response passes, so tests stay green while the extra fields quietly accumulate somewhere downstream.

I added a thin layer behind the router that records unexpected keys without touching the pass/fail decision:

# schema_drift.py — record unexpected keys instead of failing on them
import json
import logging
from collections import Counter
 
logger = logging.getLogger("schema_drift")
_unexpected: Counter = Counter()
 
 
def observe(workload: str, model: str, expected_keys: set[str], text: str) -> None:
    try:
        data = json.loads(text)
    except json.JSONDecodeError:
        logger.warning("drift/parse_error workload=%s model=%s", workload, model)
        return
    if not isinstance(data, dict):
        return
 
    extra = set(data) - expected_keys
    missing = expected_keys - set(data)
    for key in extra:
        _unexpected[(workload, model, key)] += 1
    if extra or missing:
        logger.info(
            "drift workload=%s model=%s extra=%s missing=%s",
            workload, model, sorted(extra), sorted(missing),
        )
 
 
def top_drift(limit: int = 10) -> list:
    return _unexpected.most_common(limit)

The important part is what it does not do: raise. Drift detection that can take production down is the first thing someone rips out before the next migration. Record only, and let a human review the tally weekly. Split that way, the monitoring survives.

In my case, confidence and reasoning rose to the top of the list within two weeks of the switch. Downstream code ignored unknown keys, so nothing broke — but it explained a meaningful share of that 12 percent token increase. Pass rates alone would never have surfaced it.

If you run multiple processes, an in-process Counter fragments the tally. Either route it into your existing metrics pipeline or emit the extra key names as structured logs and aggregate on the log side. I went with the latter: one extra log line, no new destination to operate.

Put Outage Fallback on the Same Router

With Gemini just recovering from what was reported as its largest outage to date (the June 12 incident with widespread error 1076 / 1099), I reviewed my fallback paths during the same migration. Once a model router exists, outage evacuation rides on the same mechanism.

# generate_with_fallback — auto-evacuate to the previous generation on failure
from model_router import FALLBACK_MODEL, resolve_model
 
 
def generate_with_fallback(client, workload, contents, config):
    primary = resolve_model(workload)
    last_error = None
    for model in (primary, FALLBACK_MODEL):
        try:
            return client.models.generate_content(
                model=model, contents=contents, config=config
            )
        except Exception as err:  # in production, catch google.genai.errors types
            last_error = err
    raise last_error

Two caveats. First, old and new models can go down in the same incident — this is insurance, not redundancy. A queue design that lets failed jobs recover the next morning matters more. Second, log every fallback activation; otherwise you get the "it silently ran on the old model for a week" accident. I send a single Slack notification whenever it triggers.

Where I Got Stuck — Aliases, Quotas, and Output Drift

Three places where the migration actually stopped my hands.

I almost shipped an alias ID. During testing I nearly specified a -latest style alias before correcting to the pinned ID. Aliases swap their contents without notice, so the model you measured and the model running in production can silently diverge. My rule: only pinned GA IDs go into the harness and the router.

Quotas are tracked per model. The first nightly batch after switching threw scattered 429s. Concurrency levels proven safe on the old model count against a separate rate-limit bucket on the new one. I halved concurrency on day one, confirmed zero 429s, then restored it.

Different thinking defaults made latency jitter. The 3.5 generation handles reasoning differently, and without explicit config the latency variance grew on some tasks. For comparative measurement, pin temperature and reasoning settings identically on both models. The cost-control mindset from Controlling thinking_budget on Gemini 2.5 Pro — cutting costs to a third while protecting reasoning quality carries over directly.

Start by Picking One Pipeline and Measuring It

The whole migration reduces to: measure, decide per workload, switch in stages through a router, and keep monitoring plus a rollback lever within reach. As the Enterprise default-lock shows, this switch will eventually arrive from the outside whether you choose it or not. I'd rather hold the decision data in my own hands while the timing is still mine.

As a next step, pick one pipeline whose output format can be verified mechanically, and run about 20 cases through the harness above. Once you have two numbers — p50 latency and format pass rate — the migration stops being a matter of intuition. I hope this record is useful to anyone doing the same work.