⬡ Advanced/2026-09-17Intermediate

The color label Gemini returned for the same wallpaper changed — and the model was not the reason

I counted color-label agreement across five ways of exporting the same wallpaper. Stripping the ICC profile was the worst of them. Here is how I pin the color space down before asking, and how I check the answers against a number I compute myself.

Gemini⁸⁷Image understandingColor spaceWallpaper appPython⁴⁷

✦ Premium Article

I was classifying the new batch of ukiyo-e wallpapers late one evening when an image that had come back as blue the week before came back as gray. The prompt had not changed. The model was the same. The one thing I had changed was the tool I used to export the files.

It took me two nights to find the cause. It was not the model having an off day — it was the image I was handing over. More precisely, I had been discarding the one hint that tells a reader which color space those pixel values belong to.

So I lined up five ways of exporting the same picture and counted. The material is 120 new images from the wallpaper app I run as an indie developer.

Only the color field was moving

The first thing I did was narrow the drift. My classifier returns four fields: subject, composition, dominant color, and usage tags. I re-read them side by side to see which one was actually changing.

Subject and composition came back identical on a second pass. So did the usage tags. The only field that moved was the dominant color.

That settled the direction. Instead of rewriting the whole prompt, I could measure one field. Narrowing the scope is why 120 images across five export variants finished in a single night.

Output rarely drifts as a whole. Count the fields separately and the thing you need to fix tends to collapse into one place.

Five export variants, defined before any measuring

The conditions

The originals are 4K images in Display P3. I produced five variants from each, with the long edge normalized to 1,024 pixels so that resolution would not get mixed into the result.

Converted to sRGB, saved as JPEG at quality 95 — this is my baseline
Left in Display P3, ICC profile embedded, JPEG at quality 95
Left in Display P3, ICC profile stripped, JPEG at quality 95
Converted to sRGB, saved as PNG
Converted to sRGB, saved as WebP at quality 80

Variant 3 is what I had actually been shipping. A profile is tens of kilobytes, and when you are trimming delivery size, it looks like free weight to drop.

The normalization itself is a few lines with Pillow's ImageCms.

# normalize_srgb.py — decide the color space yourself, before handing it over
from io import BytesIO
from PIL import Image, ImageCms
 
SRGB = ImageCms.createProfile("sRGB")
 
def to_srgb_jpeg(path: str, long_edge: int = 1024, quality: int = 95) -> bytes:
    img = Image.open(path)
    raw = img.info.get("icc_profile")
    if raw:
        # Only convert when a profile exists; converting an untagged image applies it twice
        src = ImageCms.ImageCmsProfile(BytesIO(raw))
        img = ImageCms.profileToProfile(img, src, SRGB, outputMode="RGB")
    else:
        # With no hint, sRGB is the only reasonable assumption. Log this branch
        img = img.convert("RGB")
    img.thumbnail((long_edge, long_edge), Image.LANCZOS)
    buf = BytesIO()
    # After conversion the reading is unambiguous whether or not a profile is attached
    img.save(buf, format="JPEG", quality=quality, optimize=True)
    return buf.getvalue()

profileToProfile rewrites the pixel values. Embedding a profile does not. Treating those two operations as the same thing is where my misreading started.

How the requests were sent

One image per request, same prompt, temperature pinned to 0, and the color restricted to a closed vocabulary of eight words. With free-form answers, wording differences creep in, and you can no longer tell whether the color changed or only the word for it did.

What counts as agreement

If a variant returned the same word as the baseline, I counted it as agreement. I also ran the baseline twice to measure its self-agreement. If the baseline itself drifts, there is nothing to compare against.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦You will be able to check, with 20 images from your own library, whether your export settings are changing what the model calls the color

✦You will be able to avoid the mistake that cost me two nights: the size optimization that strips metadata is the one that damages the judgment most

✦You will be able to put a closed color vocabulary and a self-computed reference value into your own pipeline as a cross-check

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

What the counting showed

Export variant	Color agreement vs. baseline	Images that disagreed	Input tokens per image
sRGB JPEG (baseline, self-agreement over two runs)	96%	5	258
P3 with ICC profile embedded, JPEG	88%	14	258
P3 pixels with the profile stripped, JPEG	71%	35	258
sRGB PNG	96%	5	258
sRGB WebP (quality 80)	93%	8	258

Variant 3 is where my expectation was inverted. Stripping the profile did worse than keeping it. At 71%, the color label changed on 35 of 120 images. The optimization I had added to make delivery lighter was doing the most damage to the judgment.

The other line worth noting is the token count: 258 for every variant. With the long edge normalized that is the expected outcome, and the fact that it is expected is the point — color accuracy here costs nothing extra.

I also looked at what kind of disagreement it was. Of those 35 images, 33 were neighboring-color swaps, moving between blue and indigo, or between orange and pink. Only two jumped to a distant color such as green to red.

Why I think this happens, offered as a guess

The Gemini API image understanding documentation covers the accepted formats and how an image becomes tokens. As far as I can read it, how ICC profiles are handled is not stated.

So what follows is my guess. Without a hint about which color space the numbers belong to, sRGB is the only reading available. Read Display P3 numbers as sRGB and the same values look more saturated than they are. Indigo drifts toward blue, pink drifts toward orange — which lines up neatly with disagreements clustered on neighboring colors.

Because it is a guess, I do not put it in the conclusion. What I can put there is the procedure for finding out which side the cause is on. Decide the color space yourself before you ask about color. That single line took my 35 images down to 5.

Rounding to a closed vocabulary, then checking against a number I compute

I would happily have stopped at normalization. But I do not always get to choose where an image came from. Client images arrive as they are, and older inventory was exported under settings I no longer use.

So I changed the asking side too. The color is restricted to eight words, and the model is allowed to answer "unknown" when it is torn.

# classify_color.py — a closed vocabulary, with "unknown" permitted
from google import genai
from google.genai import types
 
PALETTE = ["red", "orange", "yellow", "green", "blue", "indigo", "purple", "neutral", "unknown"]
client = genai.Client(api_key="YOUR_API_KEY")
 
SCHEMA = types.Schema(
    type=types.Type.OBJECT,
    properties={
        "color": types.Schema(type=types.Type.STRING, enum=PALETTE),
        "confident": types.Schema(type=types.Type.BOOLEAN),
    },
    required=["color", "confident"],
)
 
def classify(jpeg_bytes: bytes) -> dict:
    res = client.models.generate_content(
        model="gemini-3.8-flash",
        contents=[
            types.Part.from_bytes(data=jpeg_bytes, mime_type="image/jpeg"),
            # The point is not to force a "closest" color, but to let hesitation come back as hesitation
            "Pick one word for the color covering the largest area of this image. "
            "If you are torn between two colors, answer unknown.",
        ],
        config=types.GenerateContentConfig(
            temperature=0,
            response_mime_type="application/json",
            response_schema=SCHEMA,
        ),
    )
    return res.parsed  # {"color": "indigo", "confident": True}

An enum removes wording differences, so drift can only come from the color itself. Combined with the ledger I described in An agreement-rate gate approves the model swap, then your category shares move, the same comparison runs on the day you move to a newer generation.

Then the cross-check. Mean hue is something I can compute locally, so I compare it with what the model said.

# hue_check.py — a reference value of my own; only distant cases reach human eyes
import colorsys
from PIL import Image
 
HUE_BANDS = [(15, "red"), (45, "orange"), (70, "yellow"), (170, "green"),
             (240, "blue"), (265, "indigo"), (310, "purple"), (360, "red")]
 
def hue_label(path: str) -> str:
    img = Image.open(path).convert("RGB").resize((64, 64), Image.LANCZOS)
    hs, ss = [], []
    for r, g, b in list(img.getdata()):
        h, s, _ = colorsys.rgb_to_hsv(r / 255, g / 255, b / 255)
        hs.append(h * 360)
        ss.append(s)
    if sum(ss) / len(ss) < 0.12:   # low saturation cannot be described by a color name
        return "neutral"
    mean_hue = sum(hs) / len(hs)
    return next(label for edge, label in HUE_BANDS if mean_hue < edge)

Mean hue is a crude measure. It misses badly on images split evenly between two colors. That is why I never let it decide pass or fail — I use it only to route the distant cases into a queue for my own eyes. Since I switched to that, the number of images I actually look at fell to 9 out of 120.

There is one more level of rounding available. Replace color names with warm, cool, and neutral, and even the profile-stripped variant agrees 98% of the time. Neighboring-color drift does not affect a three-way split. If all you ship is a color filter in a list view, that coarseness is enough.

Three places I tripped

I thought embedding and converting were the same thing. Embedding an ICC profile leaves the pixels alone; it assumes the reader will interpret them. If you want the question settled on your side, run profileToProfile.
The profile was being dropped during thumbnail generation. The resizing library I use for delivery does not carry it over by default. Nothing raises an error. In production that silent loss is the worst kind, so I added one line at the end of the pipeline that checks Image.open(path).info.get("icc_profile") after writing.
Flattening an alpha PNG into JPEG mixes the background in. Transparent areas fill with white or black, which moves both the mean hue and the label. I built one version filled with white and one with black, compared them, and settled on white.

None of these live in the model. The habit of suspecting the steps before the request came from tripping over all three. On the resizing step itself, I have also written Why I Always Resize Images With Pillow Before Sending Them to the Gemini API.

What you can try today with 20 images

There is no need to redo all of this. Take 20 images, send two variants only — profile stripped, and converted to sRGB — and put the color labels side by side. If two or three disagree, there is room to fix things on your side of the request.

I have made that two-variant comparison the entry point whenever I adopt a new export setting. The token cost does not move, so measuring is faster than deliberating.

Thank you for reading. If it saves even one person the two nights I spent, writing it was worth the time.

Thank You for Reading

Gemini Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.