It was late, and I was sorting user-submitted wallpapers into four buckets — landscape, portrait, abstract, other. The schema listed exactly those four enum values, yet "natural" and "scenery" kept showing up in the logs. I suspected a behavior change and re-sent the same images several times. The real culprit was my own assumption: that writing a schema meant the schema would be obeyed.
A misclassification ends up as a visibly odd ordering in the app, so I wanted a fix I could verify rather than a hunch. Below are the reproducer, my reading of the cause, three layered fixes, and a small script for measuring the leak rate on your own data.
Minimal reproducer
from google import genai
from pydantic import BaseModel
from typing import Literal
MODEL = "gemini-2.5-flash" # check the official deprecations page for availability before you rely on it
class Classification(BaseModel):
label: Literal["landscape", "portrait", "abstract", "other"]
confidence: float
client = genai.Client()
response = client.models.generate_content(
model=MODEL,
contents=["Classify this wallpaper image.", image_part],
config={
"response_mime_type": "application/json",
"response_schema": Classification,
},
)
result = Classification.model_validate_json(response.text)
# Expected: label is one of the four values
# Actual: occasionally "natural" or "scenic" -> pydantic.ValidationErrorThe mismatch only showed up once I ran something close to production data. A handful of sample images will usually look fine.
Why the enum gets ignored
The official structured-output page says the output is syntactically valid JSON, but that you should always validate values in your application and prepare for outputs that match the schema yet are semantically wrong. I read that as: the schema is strong at shape, and not a promise about meaning.
How decoding works internally isn't public, so I won't pretend to know. What I can share is the pattern I noticed — a tendency, not a guarantee.
| Condition | Why it may leak (my guess) | How to check |
|---|---|---|
| Long input such as images or PDFs | The output constraint is a smaller part of what the model is attending to | Swap in a text-only input and see whether leaks drop |
| Enum values with near-synonyms | landscape vs scenic — the model reaches for a paraphrase | Check whether leaked values are synonyms of your enum values |
| No definition of the labels in the prompt | The model has to guess where each label's boundary lies | Add values and definitions, then re-measure |
I stopped here for a moment. "It feels better" is not evidence, so before fixing anything I measured.
Fix 0: measure the leak rate on your own data
This runs the same image set several times and counts how often the label falls outside the allowed values, so you can compare before and after a prompt change.
from collections import Counter
import json
ALLOWED = ["landscape", "portrait", "abstract", "other"]
def measure_drift(images, make_contents, runs: int = 3):
"""make_contents(image) -> contents. Swap it out to compare prompt variants."""
total, drift = 0, 0
bad_values = Counter()
for image in images:
for _ in range(runs):
response = client.models.generate_content(
model=MODEL,
contents=make_contents(image),
config={
"response_mime_type": "application/json",
"response_schema": Classification,
},
)
total += 1
try:
label = Classification.model_validate_json(response.text).label
except Exception:
# On ValidationError, pull the raw value so it can be counted
label = json.loads(response.text).get("label", "<missing>")
if label not in ALLOWED:
drift += 1
bad_values[label] += 1
return {"total": total, "drift": drift, "rate": drift / total, "values": bad_values.most_common(5)}Read values first. If the leaks are mostly synonyms such as scenic, Fix 1 is the right shape. If <missing> dominates, you are probably looking at a different schema violation, not an enum problem.
Fix 1: put the allowed values and their definitions in the prompt
What helped most was adding the allowed values and a definition of each label to the prompt text, on top of the schema.
prompt = f"""\
Classify the wallpaper image into exactly one of these labels:
- landscape (natural scenery, mountains, sea, sky)
- portrait (a person or animal as the main subject)
- abstract (pattern, gradient, geometric art)
- other (anything that does not fit the above)
Return JSON. The "label" field MUST be exactly one of: {ALLOWED}.
Do NOT invent new labels.
"""Stating the meaning of each label and forbidding invention leaves the model less room to guess the boundaries. Writing it twice (schema and prompt) looks redundant, but you can confirm the effect on your own data by passing lambda image: [prompt, image] to measure_drift and comparing.
Fix 2: validate on receipt, retry once
Leaks still happen, so validate the value and send a single self-correction prompt only on a mismatch. A retry loop with no cap is how costs creep up quietly.
def classify_with_retry(image_part, max_retry: int = 1) -> Classification:
last_text = ""
for attempt in range(max_retry + 1):
instruction = prompt if attempt == 0 else (
f"Your previous label '{last_text}' was not in {ALLOWED}. "
f"Re-classify using ONLY these four labels."
)
response = client.models.generate_content(
model=MODEL,
contents=[instruction, image_part],
config={
"response_mime_type": "application/json",
"response_schema": Classification,
},
)
last_text = response.text
try:
result = Classification.model_validate_json(last_text)
if result.label in ALLOWED:
return result
except Exception:
pass
# Fallback: treat as "other" so the pipeline keeps moving
return Classification(label="other", confidence=0.0)Anything that survives the retry falls back to other with confidence=0.0, which marks it as an unsure classification. That way you can find and re-run just those rows later.
Fix 3: suspect the SDK version
google-genai moves quickly, and what response_schema accepts — passing a Pydantic model directly, for instance — has changed between versions. It isn't unusual for CI or production to sit on an older version than your laptop.
pip show google-genai | grep Version
pip install --upgrade google-genaiWhen it only happens in production, compare the SDK versions across environments before touching the prompt.
Catching drift early
You can't get the leak rate to zero, so make it observable.
def log_classification(label: str):
print({
"event": "enum_classification",
"label": label,
"is_drift": label not in ALLOWED,
})Cloud Logging or BigQuery both work as a destination. When the rate jumps, it has usually been one of these three:
- The model version or snapshot changed
- The input distribution changed (a new series of images, say)
- The SDK was upgraded
Look-alike symptoms
Enum leaks resemble other schema violations, and I've mixed them up myself.
- Missing
response_mime_type→ Markdown code fences instead of JSON - Nested
oneOf→ child fields sometimes collapse to the first option - Missing
propertyOrdering→ field order drifts in long JSON
Print response.text untouched first, and tell JSON from Markdown, enum leaks from ordering issues. Ordering is covered in stabilizing field order with propertyOrdering, and how I wire validation into production is in running responseSchema validation in production.
The schema promises shape; the prompt and the receiving code own meaning. Once I drew that line, classification incidents got much calmer. A good first step: run measure_drift once over about thirty images and write down your current leak rate.