The color label Gemini returned for the same wallpaper changed — and the model was not the reason

What the counting showed

Variant 3 is where my expectation was inverted. Stripping the profile did worse than keeping it. At 71%, the color label changed on 35 of 120 images. The optimization I had added to make delivery lighter was doing the most damage to the judgment.

The other line worth noting is the token count: 258 for every variant. With the long edge normalized that is the expected outcome, and the fact that it is expected is the point — color accuracy here costs nothing extra.

I also looked at what kind of disagreement it was. Of those 35 images, 33 were neighboring-color swaps, moving between blue and indigo, or between orange and pink. Only two jumped to a distant color such as green to red.

Why I think this happens, offered as a guess

The Gemini API image understanding documentation covers the accepted formats and how an image becomes tokens. As far as I can read it, how ICC profiles are handled is not stated.

So what follows is my guess. Without a hint about which color space the numbers belong to, sRGB is the only reading available. Read Display P3 numbers as sRGB and the same values look more saturated than they are. Indigo drifts toward blue, pink drifts toward orange — which lines up neatly with disagreements clustered on neighboring colors.

Because it is a guess, I do not put it in the conclusion. What I can put there is the procedure for finding out which side the cause is on. Decide the color space yourself before you ask about color. That single line took my 35 images down to 5.

Rounding to a closed vocabulary, then checking against a number I compute

I would happily have stopped at normalization. But I do not always get to choose where an image came from. Client images arrive as they are, and older inventory was exported under settings I no longer use.

So I changed the asking side too. The color is restricted to eight words, and the model is allowed to answer "unknown" when it is torn.

# classify_color.py — a closed vocabulary, with "unknown" permitted
from google import genai
from google.genai import types
 
PALETTE = ["red", "orange", "yellow", "green", "blue", "indigo", "purple", "neutral", "unknown"]
client = genai.Client(api_key="YOUR_API_KEY")
 
SCHEMA = types.Schema(
    type=types.Type.OBJECT,
    properties={
        "color": types.Schema(type=types.Type.STRING, enum=PALETTE),
        "confident": types.Schema(type=types.Type.BOOLEAN),
    },
    required=["color", "confident"],
)
 
def classify(jpeg_bytes: bytes) -> dict:
    res = client.models.generate_content(
        model="gemini-3.8-flash",
        contents=[
            types.Part.from_bytes(data=jpeg_bytes, mime_type="image/jpeg"),
            # The point is not to force a "closest" color, but to let hesitation come back as hesitation
            "Pick one word for the color covering the largest area of this image. "
            "If you are torn between two colors, answer unknown.",
        ],
        config=types.GenerateContentConfig(
            temperature=0,
            response_mime_type="application/json",
            response_schema=SCHEMA,
        ),
    )
    return res.parsed  # {"color": "indigo", "confident": True}

An enum removes wording differences, so drift can only come from the color itself. Combined with the ledger I described in An agreement-rate gate approves the model swap, then your category shares move, the same comparison runs on the day you move to a newer generation.

Then the cross-check. Mean hue is something I can compute locally, so I compare it with what the model said.

# hue_check.py — a reference value of my own; only distant cases reach human eyes
import colorsys
from PIL import Image
 
HUE_BANDS = [(15, "red"), (45, "orange"), (70, "yellow"), (170, "green"),
             (240, "blue"), (265, "indigo"), (310, "purple"), (360, "red")]
 
def hue_label(path: str) -> str:
    img = Image.open(path).convert("RGB").resize((64, 64), Image.LANCZOS)
    hs, ss = [], []
    for r, g, b in list(img.getdata()):
        h, s, _ = colorsys.rgb_to_hsv(r / 255, g / 255, b / 255)
        hs.append(h * 360)
        ss.append(s)
    if sum(ss) / len(ss) < 0.12:   # low saturation cannot be described by a color name
        return "neutral"
    mean_hue = sum(hs) / len(hs)
    return next(label for edge, label in HUE_BANDS if mean_hue < edge)

Mean hue is a crude measure. It misses badly on images split evenly between two colors. That is why I never let it decide pass or fail — I use it only to route the distant cases into a queue for my own eyes. Since I switched to that, the number of images I actually look at fell to 9 out of 120.

There is one more level of rounding available. Replace color names with warm, cool, and neutral, and even the profile-stripped variant agrees 98% of the time. Neighboring-color drift does not affect a three-way split. If all you ship is a color filter in a list view, that coarseness is enough.

Three places I tripped

1. I thought embedding and converting were the same thing. Embedding an ICC profile leaves the pixels alone; it assumes the reader will interpret them. If you want the question settled on your side, run profileToProfile.
2. The profile was being dropped during thumbnail generation. The resizing library I use for delivery does not carry it over by default. Nothing raises an error. In production that silent loss is the worst kind, so I added one line at the end of the pipeline that checks Image.open(path).info.get("icc_profile") after writing.
3. Flattening an alpha PNG into JPEG mixes the background in. Transparent areas fill with white or black, which moves both the mean hue and the label. I built one version filled with white and one with black, compared them, and settled on white.

None of these live in the model. The habit of suspecting the steps before the request came from tripping over all three. On the resizing step itself, I have also written Why I Always Resize Images With Pillow Before Sending Them to the Gemini API.

What you can try today with 20 images

There is no need to redo all of this. Take 20 images, send two variants only — profile stripped, and converted to sRGB — and put the color labels side by side. If two or three disagree, there is room to fix things on your side of the request.

I have made that two-variant comparison the entry point whenever I adopt a new export setting. The token cost does not move, so measuring is faster than deliberating.

Thank you for reading. If it saves even one person the two nights I spent, writing it was worth the time.