The Morning Flash Started Answering Differently, Your Prompt Is Not the First Thing to Check

I was rereading the Gemini API changelog when the entry for October 8 stopped me. gemini-3.7-flash and gemini-3.5-flash were deprecated, and each would be routed automatically to a newer Flash.

Routed automatically means the model producing your replies can change even if you have not edited a single character of the model field. As an indie developer who lets Flash draft replies to store reviews, that line made me uneasy. Suppose one morning the endings of those drafts run oddly long. My first instinct would be to blame my own prompt: rewrite it, lower the temperature, rewrite it again — and only after half a day discover that the model had changed underneath me. I would rather not take that order of events.

This article is my way of reversing it. I should say up front that I have not yet confirmed which ID comes back in modelVersion after an automatic routing in my own environment. So what follows is about putting the instrument in place before the question arises.

First, keep the requested ID and the answering ID as separate columns

A generateContent response carries modelVersion, the model that actually answered (response.model_version in the Python google-genai SDK). The ID you wrote in the request and the ID that comes back are not guaranteed to be the same string.

The triage logic fits in one table.

Being able to tell row one from row two turns half a day into thirty minutes. On a day when you did not record the ID, you cannot tell which row you are in.

A three-prompt check script

Because it runs every day, I made keeping it light the top priority. Three short inputs close to everyday use, maxOutputTokens at 256, temperature fixed at 0. I am not judging prose quality. I record five things:

• The requested ID and the answering ID
• Character count
• Whether it parses as JSON (after stripping a code fence)
• finishReason
• HTTP status

import json
import os
import time
import urllib.error
import urllib.request
 
BASE = "https://generativelanguage.googleapis.com/v1beta"
REQUESTED = os.environ.get("GEMINI_MODEL", "gemini-3.7-flash")  # the model ID your own code asks for
LOG = "canary_log.jsonl"
 
# Three short inputs close to everyday use. Length and shape are all I need to see.
PROMPTS = {
    "reply_draft": "Write a polite 3-sentence reply to a 2-star review that says: too many ads.",
    "json_shape": 'Answer with JSON only. Keys: "tone" and "reason". Text: "Thanks, that really helped."',
    "short_answer": "What time of day is a wallpaper app notification most likely to be opened? One sentence.",
}
 
 
def call(prompt, key):
    body = json.dumps({
        "contents": [{"parts": [{"text": prompt}]}],
        "generationConfig": {"maxOutputTokens": 256, "temperature": 0},
    }).encode()
    req = urllib.request.Request(
        f"{BASE}/models/{REQUESTED}:generateContent",
        data=body, method="POST",
        headers={"Content-Type": "application/json", "x-goog-api-key": key},
    )
    try:
        with urllib.request.urlopen(req, timeout=30) as r:
            return r.status, json.load(r)
    except urllib.error.HTTPError as e:
        return e.code, {"error": e.read().decode()[:300]}
 
 
def summarize(label, status, resp):
    """Fold one response into a single row we can compare later."""
    cand = (resp.get("candidates") or [{}])[0]
    parts = (cand.get("content") or {}).get("parts") or [{}]
    text = parts[0].get("text", "")
    served = resp.get("modelVersion", "(none)")  # the model that actually answered
    try:
        json.loads(text.strip().removeprefix("```json").removesuffix("```").strip())
        json_ok = True
    except ValueError:
        json_ok = False
    return {
        "ts": time.strftime("%Y-%m-%dT%H:%M:%S"),
        "label": label, "status": status,
        "requested": REQUESTED, "served": served,
        "routed": served != REQUESTED and served != "(none)",
        "chars": len(text), "json_ok": json_ok,
        "finish": cand.get("finishReason", "(none)"),
    }
 
 
def last_two_runs(path=LOG):
    rows = [json.loads(l) for l in open(path, encoding="utf-8")]
    labels = sorted({r["label"] for r in rows})
    out = {}
    for lb in labels:
        mine = [r for r in rows if r["label"] == lb]
        if len(mine) >= 2:
            out[lb] = (mine[-2], mine[-1])
    return out
 
 
def report(path=LOG):
    for lb, (a, b) in last_two_runs(path).items():
        notes = []
        if a["served"] != b["served"]:
            notes.append(f"served {a['served']} -> {b['served']}")
        if a["json_ok"] != b["json_ok"]:
            notes.append(f"json_ok {a['json_ok']} -> {b['json_ok']}")
        if a["chars"] and abs(b["chars"] - a["chars"]) / a["chars"] > 0.5:
            notes.append(f"chars {a['chars']} -> {b['chars']}")
        print(f"{lb:<13}", "; ".join(notes) if notes else "no change")
 
 
if __name__ == "__main__":
    key = os.environ["GEMINI_API_KEY"]
    with open(LOG, "a", encoding="utf-8") as f:
        for label, prompt in PROMPTS.items():
            status, resp = call(prompt, key)
            row = summarize(label, status, resp)
            f.write(json.dumps(row, ensure_ascii=False) + "\n")
            print(row["label"], row["status"], row["requested"], "->", row["served"])
    report()

The key comes from the GEMINI_API_KEY environment variable, never from the command line. Put the model ID your own code uses into GEMINI_MODEL. There is no retry on purpose: if the check itself hits a 429 or 503, I would rather have that fact sit in status than be smoothed over.

From the second run onward, report() at the bottom prints only the differences from the previous run, one line per label. Days when the shape broke, when the character count fell below half or rose 1.5x, or when the answering ID changed will stand out there.

I checked the diff logic against stand-in responses

I did not call the real API for this. I built two stand-in responses, with the requested model gemini-3.7-flash and an answering model of gemini-3.8-flash, and ran them through summarize and report.

json_shape    served gemini-3.7-flash -> gemini-3.8-flash; chars 25 -> 87

The served difference marks the day the model swapped, and the chars difference shows the length moving. This output comes from data I assembled myself, so it says nothing about how the real models behave.

The order I follow on a day when something shows up

When report() prints anything, I go through these in order.

1. Did the answering ID change? If so, I leave the prompt alone and open the changelog entry for that date.
2. Is finishReason anything other than STOP? If it is MAX_TOKENS, the problem is the output cap, not the model.
3. Did JSON stop parsing? Often it is only a code fence, and stripping it in the parser is enough.
4. Did only the length move? Before adding a word limit to the instructions, I read yesterday's output next to today's, one pair.

For a proper side-by-side, a fixed question set helps. I Decide Between 3.8 Flash and 3.7 Flash With 20 Questions of My Own runs two generations on identical inputs. The script here is the step before that: finding out whether anything was swapped at all.

The answering ID also helps with cost records

Keeping the answering ID has a side benefit: you can later trace from which day billing moved to a different model's rate. Per-request token logging is covered in the usageMetadata cost-tracking article.

For handling deprecation notices, see the model deprecation migration guide. Automatic routing pushes you from doing the migration to noticing it afterward, which is why I think the check is worth turning into a routine.

The line I drew

One line came out of this exercise.

When output changes, find out which model answered before touching the prompt.

I plan to run the script once each morning from cron, with no overnight alerts. The one reading the result is me, after the coffee is made.

Start today: run the script by hand once and confirm that three rows land in canary_log.jsonl. A single day of quiet records is the baseline every later comparison stands on.