In mid-May 2026, I started reshaping the image category classification pipeline behind the iOS and Android wallpaper apps I run. The trigger was simple: Gemini 3 Pro had become noticeably more stable week by week, and I wanted to see — using my own production logs, not someone else's benchmark — exactly how much it differed from 2.5 Pro in practice.
After running wallpaper apps solo for a while, I have come to trust my own operational logs over external benchmarks when deciding whether to switch models. So instead of picking one model up front, I ran both Gemini 3 Pro and 2.5 Pro side by side for three full weeks, logging two responses for every image. This memo is what came out of that experiment.
Why parallel rather than head-to-head
Generic benchmark numbers do not predict what will happen in a specific domain. Wallpaper classification is a domain with awkward adjacent categories — photography, illustration, abstract, pattern, minimal, cosmic, nature, geometric — and the boundaries are blurry enough that even a human reviewer wavers. To claim "3 Pro is better" with any honesty, I needed two responses to the same image with the same prompt and temperature, then a human pass on the deltas.
The structure was deliberately mundane. A Python job on Cloud Run calls the google-genai SDK twice per image and lands both results in BigQuery with a model_id column. Per-request pricing is low enough that pushing roughly 110,000 images through both models over three weeks added only a few thousand yen on top of the existing bill.
from google import genai
from google.genai import types
client = genai.Client(api_key=API_KEY)
def classify(image_bytes: bytes, model: str) -> dict:
resp = client.models.generate_content(
model=model,
contents=[
types.Part.from_bytes(data=image_bytes, mime_type="image/jpeg"),
"Classify this image into one of 8 categories. "
"Return JSON: {category, confidence(0-1), reasoning(max 40 chars)}",
],
config=types.GenerateContentConfig(
response_mime_type="application/json",
temperature=0.2,
thinking_config=types.ThinkingConfig(thinking_budget=0)
if model.startswith("gemini-2.5") else None,
),
)
return resp.parsedI pinned thinking_budget=0 on the 2.5 Pro side specifically to keep both Pro tiers on a roughly comparable footing. Gemini 3 Pro inserts a small amount of thinking by default, and without the budget cap the difference would simply be the extra cognition rather than the model itself.
Three things three weeks of logs surfaced
Reviewing the logs across accuracy, cost, and latency, three differences stood out.
The first was boundary-category robustness. On overlapping cases such as "minimal yet natural" or "geometric yet abstract," Gemini 3 Pro produced a more stable first-choice category. With 2.5 Pro, confidence on those edge cases tended to cluster between 0.55 and 0.7, and roughly 13% of those images ended up queued for manual review. With 3 Pro alone, the same review-queue rate was about 7%.
The second was the quality of the reasoning field. This is not a number — it is a feel — but 3 Pro tended to return short observational notes like "soft blue coverage with central whitespace," while 2.5 Pro leaned toward word lists like "minimal" and "abstract." Since I surface the reasoning string in the internal moderation dashboard, 3 Pro's wording lets the operator decide faster.
The third was perceived latency. Per-image p50 latency was around 1.8 seconds for 2.5 Pro and around 2.1 seconds for 3 Pro — roughly 15 to 20% slower. For batch use I parallelise 20 jobs at a time so the difference disappears, but on user-interactive flows I still see room for keeping 2.5 Pro Flash in the rotation.
What broke, and what I worked around
Halfway through the experiment, I had to redo the tiebreaker layer. My first version simply picked whichever response had the higher confidence value. That worked in theory but kept favouring 2.5 Pro because 3 Pro tends to report somewhat lower confidence numbers even when its reasoning is clearly stronger.
This started leaking into product: the lineup of themed locked folders that users unlock through AdMob rewarded video began to drift toward 2.5 Pro's choices, and by the end of the second week I could feel that the themed sets were less coherent than I wanted them to be.
The fix was a domain-specific rule: when the confidence gap is below 0.15, prefer 3 Pro; otherwise honor the numbers. It is not a universal rule, but on a sampled 500-image review, the model's decision matched my human judgment 91% of the time after this change.
Writing this tiebreaker layer reminded me that a model's standing on a general-purpose benchmark and the tuning that actually pays off in your own domain are two different things. In a niche as narrow as wallpaper, weaving a model's confidence quirks into an operational rule mattered far more than a few points of difference on a public score. Even the confidence-gap threshold (0.15 here) was not something I could set up front — I nudged it into place by checking it against a 500-image sample review.
How the table is shaped on the logging side
The single biggest enabler of this comparison was logging the results at a granularity that survives later analysis. The BigQuery table I ended up with is minimal:
CREATE TABLE wallpaper_classification (
image_id STRING NOT NULL,
model_id STRING NOT NULL, -- gemini-3-pro / gemini-2.5-pro
category STRING NOT NULL,
confidence FLOAT64 NOT NULL,
reasoning STRING,
latency_ms INT64,
cost_usd NUMERIC,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);The composite of image_id and model_id acts as a logical primary key, so every image always has exactly two rows once both models have processed it. I included the cost_usd column because I wanted to know, by week three, what classifying a single image actually cost end to end, with thinking tokens included.
The measured numbers: about 0.0026 USD per image for 2.5 Pro and about 0.0034 USD for 3 Pro. Per-call, 3 Pro is roughly 30% more expensive, but the drop in human review rate from 13% to 7% saves more labour than the unit-cost gap, so the overall economics improve. For the AdMob-rewarded themed folders, classification accuracy translates directly into themed-lineup coherence, which matters more than the unit price suggests.
Where this goes next
After three weeks, the plan I settled on is "3 Pro as the primary, 2.5 Pro as a shadow."
For now, the production job switches to 3 Pro alone, and 2.5 Pro runs as a 10% weekly shadow batch. Even when 3 Pro is doing the actual work, keeping the 2.5 Pro log alive gives me an audit trail for explaining decisions later and a ready-made regression detector when Gemini issues an upstream update.
The one habit I try to keep in tooling decisions is to not stop observing the moment something feels settled. Choosing 3 Pro as the primary today does not mean halting observation of 2.5 Pro Flash and other variants — keeping their logs running is exactly what catches regressions when a Gemini-side update shifts behaviour, and that margin pays off for an indie developer who plans to operate for a long time.
The next experiment is to extend the same parallel-logging structure to a recommendation engine that pairs Gemini 3 Pro with Imagen 4, going from image to semantic tag to candidate image. I will keep deciding by the numbers from the field.
Thank you for reading along.