How I Decide Between 3.8 Flash and 3.7 Flash — A Fixed Set of 20 Questions

The change was one line. I swapped gemini-3.7-flash for gemini-3.8-flash in a small call that tidies up category blurbs for the Lab sites, let it run overnight, and read the output the next morning. The sentences were put together differently than the day before, for the same inputs.

I could not tell which version was better. Side by side, the newer one looked more careful. But careful and short are not the same thing, and short was what I had asked for.

And the bigger problem: I no longer had yesterday's output. I had nothing to compare against.

The docs say 3.7 Flash is still fine

The first thing worth saying is that moving to a new generation is not an obligation.

The What's new in Gemini 3.8 Flash page describes the model as built for long-horizon software engineering, autonomous agents, and complex enterprise workflows. It takes smaller reasoning steps, calls tools iteratively, and verifies its own work along the way — and it can spend more tokens doing so, by design.

Then comes the sentence I keep going back to: not every workflow needs that level of verification. For everyday tasks you can lower the reasoning effort, or stay on 3.7 Flash, which remains fully supported.

So the official material takes you as far as "there are more dials now." Which setting suits your own work is something you have to find out on your machine.

Level the ground before you compare

I got this wrong the first time, so I will write it down.

The migration checklist for 3.8 Flash asks you to strip temperature, top_p, and top_k from your generation config, replace thinking_budget with the string enum thinking_level, and drop candidate_count. I did exactly that on the 3.8 side. And I left the 3.7 side running with its existing config.

That did not work out. Differences showed up, but I could no longer tell whether they came from the generation change or from the temperature=0.2 I had just removed.

Obvious in hindsight. Rebuild the comparison around the older setup, not the newer one. Strip the sampling parameters from the 3.7 side too, get both to a plain state, and only then line them up. Skip this and every number downstream becomes unreadable.

Build the question set from things that were already wrong

The second rule I follow: do not invent new questions.

Questions you think up tend to be questions you already know the answer to. What you want instead are the outputs you have already gone back and fixed — the sentence a reviewer put back, the paragraph you cut before publishing, the line in an app store description you rewrote because the tone was off. In my case, twenty of those filled the file almost immediately.

I keep it as JSONL, one question per line. Since I want the grading to be mechanical, each line carries words that should appear and words that should not.

{"id": "cat-desc-01", "input": "Write one sentence describing the Japanese-style category in a wallpaper app.", "must_include": ["Japanese"], "must_not_include": ["best ever", "don't miss"]}
{"id": "cat-desc-02", "input": "Rewrite that same sentence using exactly one full stop.", "must_include": ["."], "must_not_include": ["!"]}

The must_not_include side does more work than you would expect. What grows when you move up a generation is rarely wrongness — it is decoration you never asked for.

Run each question three times, against both generations

A single run per question cannot tell you whether the model is inconsistent or whether you are. So I run each question three times and count a question as stable only when all three verdicts agree.

import json
import statistics
import time
 
from google import genai
 
client = genai.Client()
 
MODELS = [
    {"name": "gemini-3.7-flash", "config": {}},
    {"name": "gemini-3.8-flash", "config": {"thinking_level": "medium"}},
]
REPEAT = 3
SYSTEM = "Answer in under 60 words. Do not use bullet points."
 
 
def ask(model, config, prompt):
    started = time.time()
    interaction = client.interactions.create(
        model=model,
        input=prompt,
        system_instruction=SYSTEM,
        generation_config=config,
    )
    usage = interaction.usage
    return {
        "text": interaction.output_text,
        "out_tokens": usage.total_output_tokens or 0,
        "thought_tokens": usage.total_thought_tokens or 0,
        "seconds": round(time.time() - started, 2),
    }
 
 
def judge(text, item):
    hit = all(word in text for word in item.get("must_include", []))
    miss = any(word in text for word in item.get("must_not_include", []))
    return hit and not miss
 
 
def run(path):
    with open(path, encoding="utf-8") as handle:
        items = [json.loads(line) for line in handle if line.strip()]
 
    for model in MODELS:
        passed = agreed = 0
        out_tokens, thoughts, seconds = [], [], []
 
        for item in items:
            runs = [ask(model["name"], model["config"], item["input"]) for _ in range(REPEAT)]
            verdicts = [judge(run_result["text"], item) for run_result in runs]
            passed += sum(verdicts)
            agreed += 1 if len(set(verdicts)) == 1 else 0
            out_tokens += [r["out_tokens"] for r in runs]
            thoughts += [r["thought_tokens"] for r in runs]
            seconds += [r["seconds"] for r in runs]
 
        total = len(items) * REPEAT
        print(
            model["name"],
            f"passed {passed}/{total}",
            f"stable {agreed}/{len(items)}",
            f"median output tokens {statistics.median(out_tokens):.0f}",
            f"median thought tokens {statistics.median(thoughts):.0f}",
            f"median seconds {statistics.median(seconds):.2f}",
        )
 
 
if __name__ == "__main__":
    run("eval_set.jsonl")

The four values pulled from usage each do a different job. total_output_tokens is what you are billed for, total_thought_tokens is what the model spent thinking, and the elapsed seconds are what your users actually feel. I report medians rather than means because one long-running question out of twenty will drag a mean around.

It is worth running the whole thing once more with a different thinking_level. For me, 3.8 Flash at low turned out to be the column closest to how I actually work.

Five rows are enough to read the result

Across my twenty questions the pass rate barely moved, stability held, and only the output token count climbed noticeably. That made it a cost question rather than a quality one, so I re-measured with thinking_level set to low and got most of the increase back. I wrote separately about why the same per-token price can still raise your bill.

Two questions did lose stability, though. Both were ones where my own instruction had been vague. I set out to test a model generation and ended up finding the loose edges in my own prompts.

Where I draw the line

Pin the model ID, and only move it on a day the question set passes. That order is the one thing I try not to bend, even when I am in a hurry.

I feel the pull to upgrade on release day as much as anyone. But upgrading that day gives you the sensation of being current and nothing else — you still cannot say whether your actual work got better. Run the twenty questions first and you can name both what improved and what regressed.

Start with five. Think of an output you once went back and corrected, and put the pre-correction wording into must_not_include. That alone will make the morning after the next release far easier.

Thank you for reading. My own set is still being shaped — twenty is simply where it sits this week.