◈ API / SDK/2026-07-16Advanced

I Asked Gemini to Grade My App Store Screenshots. Everything Scored 78–85.

Ask Gemini Vision to grade App Store screenshots out of 100 and good candidates and deliberately broken ones both land at 78–85. Here is how I measured the judge's discrimination power, dropped absolute scoring, and rebuilt it as debiased pairwise comparison.

Gemini API¹⁸⁷ Vision⁴ ASO² App Store⁶ LLM-as-judge Pairwise Comparison Python³⁸ Indie Dev⁷

✦ Premium Article

Before replacing all five of my App Store screenshots, I asked Gemini to grade the new set. It came back with 82. I fed it the old set as a control: 79.

Three points apart. That told me nothing, so I built a deliberately broken version — every headline erased, contrast crushed, panels shuffled — and fed that in too.

If a set with zero readable copy earns a 76, then 82 means nothing either. That was the day I stopped asking Vision for a score.

What follows is how I proved the grader was unusable at my decision granularity, and what I replaced it with. As an indie developer who has rebuilt a wallpaper app's store listing more times than I'd like to admit, this was the single change that made the tooling worth running.

The minimum setup for doubting your own grader

pip install google-genai pillow --break-system-packages

import os
import json
import statistics
from pathlib import Path
from itertools import combinations
 
from google import genai
from google.genai import types
 
client = genai.Client(api_key=os.environ["GOOGLE_API_KEY"])
 
# Verification burns a lot of calls, so Flash. Only the final head-to-head goes to Pro.
JUDGE_MODEL = "gemini-flash-latest"
 
 
def load_image(path: str) -> types.Part:
    """Load a local image into a Part, inferring MIME from the extension."""
    suffix = Path(path).suffix.lower()
    mime = {
        ".jpg": "image/jpeg",
        ".jpeg": "image/jpeg",
        ".png": "image/png",
        ".webp": "image/webp",
    }.get(suffix, "image/jpeg")
 
    with open(path, "rb") as f:
        return types.Part.from_bytes(data=f.read(), mime_type=mime)

One caveat on that model string. gemini-flash-latest is an alias, and as of July 2026 it resolves to Gemini 3.5 Flash. If you run scheduled jobs against an alias, the thing doing your grading can change underneath you without a single line of your code moving. Pin down what the alias currently points at before you trust any measurement you take today.

The grader itself is conventional. The thing under suspicion is not the function — it's the numbers coming out of it.

def score_absolute(image_paths: list[str], app_name: str, market: str = "United States") -> dict:
    """Grade a screenshot set from 0-100 (the conventional absolute approach)."""
    parts = [
        types.Part.from_text(
            f"""You are an App Store optimization expert.
Evaluate the following screenshot set.
 
App: {app_name}
Target market: {market}
 
Return exactly this JSON:
{{
  "overall_score": <integer 0-100>,
  "message_clarity": <0-100>,
  "visual_hierarchy": <0-100>,
  "reason": "one sentence justifying the score"
}}"""
        )
    ]
    for i, p in enumerate(image_paths):
        parts.append(types.Part.from_text(f"Screenshot {i + 1}:"))
        parts.append(load_image(p))
 
    resp = client.models.generate_content(
        model=JUDGE_MODEL,
        contents=parts,
        config=types.GenerateContentConfig(
            response_mime_type="application/json",
            temperature=0.2,
        ),
    )
    return json.loads(resp.text)

Measure the noise first: ten runs, one image

You cannot judge a grader from a single output. Send the same input repeatedly and find out how much the answer moves on its own.

def measure_noise(image_paths: list[str], app_name: str, n: int = 10) -> dict:
    """Grade identical input n times and measure the spread."""
    scores = []
    for i in range(n):
        result = score_absolute(image_paths, app_name)
        scores.append(result["overall_score"])
        print(f"  run {i + 1}: {result['overall_score']}")
 
    return {
        "scores": scores,
        "mean": statistics.mean(scores),
        "stdev": statistics.stdev(scores) if len(scores) > 1 else 0.0,
        "range": max(scores) - min(scores),
    }

At temperature=0.2 I expected near-identical numbers. On my set — five portrait screenshots, wallpaper category — here is what came back.

Metric	Measured
Mean	81.4
Standard deviation	2.3
Min–max	78–85 (7-point spread)

Seven points of movement on an unchanged image. Which means the 3-point gap between my new set and my old set was sitting entirely inside the noise floor.

Dropping to temperature=0 only narrowed the spread to 3 points. Multimodal inputs carry variance in tiling and tokenization that temperature does not reach. This is not a problem you turn down the temperature to fix, so I stopped trying.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦A repeatable way to decide whether an LLM judge is usable at all: measure run-to-run noise, then divide your signal gap by it

✦Ablation code that deliberately degrades your own screenshots to test what the grader can and cannot detect

✦Full pairwise comparison implementation with order-swap debiasing, plus win-rate ranking and a token cost breakdown

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

Measure the signal: break your own images on purpose

With the noise floor known, the next question is how large a real difference has to be before the grader notices. Build a control group by deliberately degrading the originals — an ablation study, pointed at the judge instead of the model.

from PIL import Image, ImageEnhance
 
 
def make_degraded_variants(src_path: str, out_dir: str) -> dict[str, str]:
    """Generate intentionally degraded images as a control group.
 
    If the grader cannot detect damage this obvious, it cannot detect anything subtler.
    """
    out = Path(out_dir)
    out.mkdir(parents=True, exist_ok=True)
    img = Image.open(src_path).convert("RGB")
    stem = Path(src_path).stem
    variants = {}
 
    # (1) Crush contrast to 30% - legibility damage
    low_contrast = ImageEnhance.Contrast(img).enhance(0.3)
    p = out / f"{stem}_lowcontrast.png"
    low_contrast.save(p)
    variants["low_contrast"] = str(p)
 
    # (2) Paint over the top third - erases the headline zone
    no_text = img.copy()
    w, h = no_text.size
    no_text.paste((242, 242, 242), (0, 0, w, h // 3))
    p = out / f"{stem}_notext.png"
    no_text.save(p)
    variants["no_text"] = str(p)
 
    # (3) Downscale hard and upscale back - resolution damage
    blurred = img.resize((w // 8, h // 8)).resize((w, h))
    p = out / f"{stem}_lowres.png"
    blurred.save(p)
    variants["low_res"] = str(p)
 
    return variants

Then divide the score gap by the noise you just measured. That ratio is the grader's actual resolving power.

def discrimination_margin(
    original: list[str],
    degraded: list[str],
    app_name: str,
    noise_stdev: float,
    n: int = 5,
) -> dict:
    """margin = (original score - degraded score) / noise stdev.
 
    Below 1.0, the gap is drowned in noise and carries no information.
    """
    orig_scores = [score_absolute(original, app_name)["overall_score"] for _ in range(n)]
    deg_scores = [score_absolute(degraded, app_name)["overall_score"] for _ in range(n)]
 
    gap = statistics.mean(orig_scores) - statistics.mean(deg_scores)
    return {
        "original_mean": statistics.mean(orig_scores),
        "degraded_mean": statistics.mean(deg_scores),
        "gap": gap,
        "margin": gap / noise_stdev if noise_stdev > 0 else float("inf"),
        "usable": (gap / noise_stdev) >= 2.0 if noise_stdev > 0 else True,
    }

My numbers:

Degradation	Mean score	Gap vs original	Margin
Original	81.4	—	—
Contrast at 30%	74.8	6.6	2.9
Headline erased	76.2	5.2	2.3
Resolution at 1/8	71.0	10.4	4.5

This is where my read of the situation changed. The grader was not broken. Show it visibly ruined images and it marks them down with a margin of 2 to 4 times the noise. It works.

The problem was that nothing I actually needed to decide lived in that range. A 14-character headline versus a 22-character one. A lifestyle photo in slot one versus a UI capture. Those differences are nowhere near "contrast crushed to 30%." When I ran my five real candidates through it, the widest gap was 4 points — a margin of 1.7, under my 2.0 threshold.

So absolute scoring could only detect differences I already knew the answer to. A grader that confirms the image I broke is broken is not a grader I can make decisions with.

Why scores collapse toward the middle

Thinking it through, the fault was in my prompt, not the model.

"Rate this from 0 to 100" gives the model no anchor. Eighty compared to what? With no reference point defined, the model settles into the safest band its training distribution offers — for this task, 70 to 85. Zero and 100 almost never have defensible justifications, so they never appear.

This is a well-documented property of LLM-as-judge setups, and nothing about it is specific to Vision. Text quality scoring collapses the same way. I had simply assumed images would be different.

If the missing piece is a reference point, supply one — put the thing you're comparing against directly into the input.

Pairwise: ask only who wins

Drop the number. Show two candidates side by side and ask for a winner.

def pairwise_compare(
    a_paths: list[str],
    b_paths: list[str],
    app_name: str,
    market: str = "United States",
) -> dict:
    """Compare two screenshot candidates and return a winner."""
    parts = [
        types.Part.from_text(
            f"""Compare two App Store screenshot candidates.
 
App: {app_name}
Target market: {market}
 
Judge on one criterion only: which set makes a user scrolling the store
more likely to want this app within the first second?
Judge on likelihood of download, not on design craft.
 
Return exactly this JSON:
{{
  "winner": "A" | "B" | "tie",
  "confidence": "high" | "medium" | "low",
  "reason": "the specific difference that decided it (1-2 sentences)"
}}"""
        )
    ]
 
    parts.append(types.Part.from_text("=== Candidate A ==="))
    for p in a_paths:
        parts.append(load_image(p))
 
    parts.append(types.Part.from_text("=== Candidate B ==="))
    for p in b_paths:
        parts.append(load_image(p))
 
    resp = client.models.generate_content(
        model=JUDGE_MODEL,
        contents=parts,
        config=types.GenerateContentConfig(
            response_mime_type="application/json",
            temperature=0.3,
        ),
    )
    return json.loads(resp.text)

This introduces a new trap: position bias. Models tend to favor whichever candidate came first (or last) regardless of content.

The fix is cheap. Ask again with the order flipped, and only count a win when both orderings agree.

def compare_debiased(a_paths, b_paths, app_name) -> str:
    """Compare twice with swapped order to cancel position bias.
 
    Returns "A" / "B" / "tie". Disagreement between the two runs counts as a tie.
    """
    first = pairwise_compare(a_paths, b_paths, app_name)["winner"]
 
    # Swap the slots, then translate the answer back into A/B terms
    swapped = pairwise_compare(b_paths, a_paths, app_name)["winner"]
    second = {"A": "B", "B": "A", "tie": "tie"}[swapped]
 
    if first == second:
        return first
    return "tie"

Across 30 pairs, 18% disagreed when I flipped the order. That 18% is exactly the set of verdicts I would have accepted at face value had I asked only once. The flip side is more encouraging: in 82% of pairs, the same candidate won no matter which slot it sat in. Absolute scoring never gave me that kind of stability.

Turn win rates into a ranking

The rest is round robin.

def rank_candidates(candidates: dict[str, list[str]], app_name: str) -> list[dict]:
    """Compare every candidate against every other, then rank by win rate.
 
    candidates: {"candidate name": [screenshot paths, ...]}
    """
    names = list(candidates.keys())
    wins = {n: 0.0 for n in names}
    matches = {n: 0 for n in names}
    log = []
 
    for a, b in combinations(names, 2):
        winner = compare_debiased(candidates[a], candidates[b], app_name)
        matches[a] += 1
        matches[b] += 1
 
        if winner == "A":
            wins[a] += 1.0
        elif winner == "B":
            wins[b] += 1.0
        else:  # tie
            wins[a] += 0.5
            wins[b] += 0.5
 
        log.append({"a": a, "b": b, "winner": winner})
        print(f"  {a} vs {b} -> {winner}")
 
    ranked = [
        {
            "name": n,
            "win_rate": round(wins[n] / matches[n], 3) if matches[n] else 0.0,
            "wins": wins[n],
            "matches": matches[n],
        }
        for n in names
    ]
    ranked.sort(key=lambda x: x["win_rate"], reverse=True)
    return ranked

Five candidates means ten pairs, twenty requests with the swap. Here is what came out.

Candidate	Win rate	Absolute score (for reference)
C (lifestyle photo in slot 1)	0.875	81
A (current, UI in slot 1)	0.625	82
E (headline cut to 14 chars)	0.500	80
B (recolored)	0.375	83
D (trimmed to 3 screenshots)	0.125	79

Look at the right-hand column. Last place scores 79, first place scores 81 — two points. Same images, same model, and only the question changed, yet the win rates spread from 0.125 to 0.875, a seven-fold gap. The information was in the model the whole time. My question was what failed to extract it.

Does round robin stay affordable?

Pair count grows quadratically, so estimate before you run.

Gemini bills an image at a flat 258 tokens if it fits within 768×768, and tiles it beyond that with tokens per tile. App Store screenshots are tall (1290×2796 and similar), so sending them untouched inflates tile counts.

I downscale to 828px wide first. ASO judgment needs overall composition and headline legibility, not pixel fidelity.

Item	Estimate
Images per candidate	5
Images per pair comparison	10 (5 for A, 5 for B)
Round robin over 5 candidates	10 pairs × 2 swap runs = 20 requests
Total image inputs	200

Flash handled the comparisons fine. I only escalate the final pair — the top two — to Pro, to get a carefully argued rationale. Running everything on Pro costs more than the accuracy difference is worth.

There is a shortcut when the field grows. Instead of round robin, fix your current live set as a permanent opponent and run each candidate against it once — a challenger format. Five candidates becomes 4 pairs × 2 = 8 requests. You lose the full ordering, but when the only question is "did anything beat what's already shipping," that's enough. I switch to it once I have more than six candidates.

The limits I have not resolved

What this measures is which set Gemini prefers, not which set users download. I have not verified the correlation between win rate and actual conversion rate. Doing so means accumulating enough samples through App Store Connect's product page optimization, and at indie scale that takes months.

So let me be precise about what this is: not a replacement for A/B testing, but the filter that decides what goes into the A/B test. Product page optimization lets you run three treatments at once. When you have seven candidates, something has to cut it to three — and that step used to be my eye and my mood. Now it's a procedure I can rerun and inspect. That's all it is. It was enough to be worth building.

One more thing. Swapping the market changes the verdicts. Running the same five candidates with market="Japan" versus market="United States" reordered my top two. Google's own Southeast Asia report notes roughly 70% of Gemini app prompts in the region are in local languages — a reminder that markets are not interchangeable defaults. Leaving the market parameter off doesn't make the judgment neutral; it silently picks one for you. Don't leave it off.

Running this tomorrow

Cut to five candidates or fewer (round robin's practical ceiling)
Run measure_noise on identical input — ten calls, a few minutes
Build degraded variants and compute the discrimination margin. Under 2.0, absolute scoring can't see differences at your granularity
Run rank_candidates for win rates. Do not skip the order swap
Send the top two to App Store Connect product page optimization
When real data arrives, record whether the win rate predicted it

Keep doing step six long enough and you can finally answer whether Gemini's win rate correlates with real conversion. I'm not there yet. When I get there, I'll publish the number.

When I first saw that 82, I nearly believed it. Numbers reassure us whether or not they've earned it. Build a grader, then grade the grader — it looks like a detour, and it's the shortest path there is.

Thanks for reading. I hope it saves you a few weeks of trusting the wrong number.

Thank You for Reading

Gemini Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.