I was running a batch that sorts wallpaper submissions into 30 categories, and I fully expected responseSchema to occasionally return a string outside its enum. What I did not expect was that retrying those outliers would mostly return the same word a second time. An image I wanted labeled "landscape" came back as "scenery", and throwing it again returned "scenery". The model was not broken. For that image, it genuinely believed that word was the closest fit.
So this was a vocabulary mismatch, not a random glitch. Hitting a mismatch with retries only adds round trips and wait time while the answer stays the same. As an indie developer who runs these batches overnight, every extra round trip shows up directly in the next morning's completion time. What follows is a field note: measure how the labels miss first, then see what actually changes when you slip a normalization layer in ahead of any retry.
I covered why the enum gets ignored, and the first-line workaround, in an earlier piece on why responseSchema returns strings outside the enum. This is the sequel: it digs into why the straightforward "validate and retry once" stops paying off once the volume grows.
Before suspecting retries, I looked at how labels miss
Before deciding on a fix, I ran the whole set through once and simply counted how the out-of-enum labels were missing. Acting on a vague sense that "it sometimes returns odd values" makes it impossible to pick the right move.
The set was 8,142 wallpapers, classified in a single pass with a 30-category enum in responseSchema. 312 results (about 3.8%) failed to match the enum. Going through those 312 by hand, the misses split cleanly into three piles.
| How it missed | Count | Example (returned value to intended label) |
|---|---|---|
| Alias / spelling variation (mechanically resolvable) | 231 | scenery to landscape, night view to nightscape, flower to plant |
| Semantically close but not in the alias table | 49 | "a photo of a canyon" to nature, "city lights" to nightscape |
| Genuinely ambiguous / multiple matches | 32 | compound "nature and animal", high-abstraction cases to hold |
231, roughly 74%, were cases where the model's word and my ground-truth label were simply saying the same thing in different words. The moment that landed, retrying revealed itself as the wrong idea. Resubmit the same image and the model's sense of vocabulary is unchanged, so "scenery" comes back again. What needed fixing was the receiving side, not the number of calls to the model.
The other thing that mattered was the 32 in the bottom row. That group must not be normalized. Snapping each one to its nearest label pushes images that belong in "other," or in a human review queue, into a plausible-looking category. Ironically, that push is what erodes trust in the classifier more than an empty label ever does.
Resolving near-misses mechanically, in two stages
The approach is simple. When an out-of-set label arrives, look it up in a deterministic alias table first; if that misses, snap it to the nearest category by embedding; and if you still lack confidence, leave it un-normalized and route it to hold. A re-query to the model is reserved only for the held items that are worth asking again from a different angle.
The alias table is built only from outliers I have actually observed. Trying to cover every case from imagination produces a dictionary nobody can maintain, so I grow it slowly, wiring words that show up in the deviation log to a canonical label after confirming them.
import unicodedata
from google import genai
from google.genai import types
client = genai.Client(api_key="YOUR_API_KEY")
# the ground-truth vocabulary (an excerpt of the 30 categories)
CANONICAL = ["landscape", "nightscape", "plant", "animal", "nature", "abstract", "space", "sea", "other"]
# wire only words actually seen in the deviation log; never add from imagination
ALIASES = {
"scenery": "landscape", "けしき": "landscape", "風景": "landscape",
"night view": "nightscape", "夜景": "nightscape",
"flower": "plant", "花": "plant", "plants": "plant",
"wildlife": "animal", "nature scene": "nature",
}
def _norm(s: str) -> str:
# absorb width, case, and surrounding-space variation before lookup
return unicodedata.normalize("NFKC", s).strip().lower()
_ALIAS_INDEX = {_norm(k): v for k, v in ALIASES.items()}Anything the first-stage alias table resolves is accepted straight as canonical. This handles the bulk of those 231 cases.
The second stage snaps words not in the alias table (the group of 49) to their nearest category with gemini-embedding-2. The key is not to decide on the distance to the top category alone, but to look at the gap (margin) between first and second place. When the margin is small, the case is genuinely a toss-up, so I drop it to hold rather than force a fit.
import numpy as np
def _embed(texts: list[str]) -> np.ndarray:
resp = client.models.embed_content(
model="gemini-embedding-2",
contents=texts,
config=types.EmbedContentConfig(task_type="SEMANTIC_SIMILARITY"),
)
vecs = np.array([e.values for e in resp.embeddings], dtype=np.float32)
# normalize to unit vectors so we can compare with cosine similarity
return vecs / np.linalg.norm(vecs, axis=1, keepdims=True)
# build the canonical vectors once at startup and reuse them
_CANON_VECS = _embed(CANONICAL)
def nearest_canonical(label: str, margin: float = 0.06):
v = _embed([label])[0]
sims = _CANON_VECS @ v # similarity to each category
order = np.argsort(sims)[::-1]
top, second = order[0], order[1]
if sims[top] - sims[second] < margin:
return None, float(sims[top]) # first and second too close, do not snap
return CANONICAL[top], float(sims[top])I fold both stages into a single acceptance gate. The return value always carries the route by which it was resolved, so it can be logged later. Without a visible route, you cannot tell whether to grow the alias table or tune the margin.
def resolve_label(raw: str):
if raw in CANONICAL:
return raw, "in_set"
hit = _ALIAS_INDEX.get(_norm(raw))
if hit:
return hit, "alias"
canon, sim = nearest_canonical(raw)
if canon is not None and sim >= 0.82:
return canon, f"embedding:{sim:.3f}"
# anything reaching here is not settled mechanically; send it to hold
return None, "unresolved"Only the images for which resolve_label returns None become candidates for human review or a re-query from a different angle. That is the first point where a limited resubmission to the model earns its keep. It is a completely different round-trip count from retrying everything unconditionally.