I Asked Gemini to Grade My App Store Screenshots. Everything Scored 78–85.

Measure the signal: break your own images on purpose

With the noise floor known, the next question is how large a real difference has to be before the grader notices. Build a control group by deliberately degrading the originals — an ablation study, pointed at the judge instead of the model.

from PIL import Image, ImageEnhance
 
 
def make_degraded_variants(src_path: str, out_dir: str) -> dict[str, str]:
    """Generate intentionally degraded images as a control group.
 
    If the grader cannot detect damage this obvious, it cannot detect anything subtler.
    """
    out = Path(out_dir)
    out.mkdir(parents=True, exist_ok=True)
    img = Image.open(src_path).convert("RGB")
    stem = Path(src_path).stem
    variants = {}
 
    # (1) Crush contrast to 30% - legibility damage
    low_contrast = ImageEnhance.Contrast(img).enhance(0.3)
    p = out / f"{stem}_lowcontrast.png"
    low_contrast.save(p)
    variants["low_contrast"] = str(p)
 
    # (2) Paint over the top third - erases the headline zone
    no_text = img.copy()
    w, h = no_text.size
    no_text.paste((242, 242, 242), (0, 0, w, h // 3))
    p = out / f"{stem}_notext.png"
    no_text.save(p)
    variants["no_text"] = str(p)
 
    # (3) Downscale hard and upscale back - resolution damage
    blurred = img.resize((w // 8, h // 8)).resize((w, h))
    p = out / f"{stem}_lowres.png"
    blurred.save(p)
    variants["low_res"] = str(p)
 
    return variants

Then divide the score gap by the noise you just measured. That ratio is the grader's actual resolving power.

def discrimination_margin(
    original: list[str],
    degraded: list[str],
    app_name: str,
    noise_stdev: float,
    n: int = 5,
) -> dict:
    """margin = (original score - degraded score) / noise stdev.
 
    Below 1.0, the gap is drowned in noise and carries no information.
    """
    orig_scores = [score_absolute(original, app_name)["overall_score"] for _ in range(n)]
    deg_scores = [score_absolute(degraded, app_name)["overall_score"] for _ in range(n)]
 
    gap = statistics.mean(orig_scores) - statistics.mean(deg_scores)
    return {
        "original_mean": statistics.mean(orig_scores),
        "degraded_mean": statistics.mean(deg_scores),
        "gap": gap,
        "margin": gap / noise_stdev if noise_stdev > 0 else float("inf"),
        "usable": (gap / noise_stdev) >= 2.0 if noise_stdev > 0 else True,
    }

My numbers:

This is where my read of the situation changed. The grader was not broken. Show it visibly ruined images and it marks them down with a margin of 2 to 4 times the noise. It works.

The problem was that nothing I actually needed to decide lived in that range. A 14-character headline versus a 22-character one. A lifestyle photo in slot one versus a UI capture. Those differences are nowhere near "contrast crushed to 30%." When I ran my five real candidates through it, the widest gap was 4 points — a margin of 1.7, under my 2.0 threshold.

So absolute scoring could only detect differences I already knew the answer to. A grader that confirms the image I broke is broken is not a grader I can make decisions with.

Why scores collapse toward the middle

Thinking it through, the fault was in my prompt, not the model.

"Rate this from 0 to 100" gives the model no anchor. Eighty compared to what? With no reference point defined, the model settles into the safest band its training distribution offers — for this task, 70 to 85. Zero and 100 almost never have defensible justifications, so they never appear.

This is a well-documented property of LLM-as-judge setups, and nothing about it is specific to Vision. Text quality scoring collapses the same way. I had simply assumed images would be different.

If the missing piece is a reference point, supply one — put the thing you're comparing against directly into the input.

Pairwise: ask only who wins

Drop the number. Show two candidates side by side and ask for a winner.

def pairwise_compare(
    a_paths: list[str],
    b_paths: list[str],
    app_name: str,
    market: str = "United States",
) -> dict:
    """Compare two screenshot candidates and return a winner."""
    parts = [
        types.Part.from_text(
            f"""Compare two App Store screenshot candidates.
 
App: {app_name}
Target market: {market}
 
Judge on one criterion only: which set makes a user scrolling the store
more likely to want this app within the first second?
Judge on likelihood of download, not on design craft.
 
Return exactly this JSON:
{{
  "winner": "A" | "B" | "tie",
  "confidence": "high" | "medium" | "low",
  "reason": "the specific difference that decided it (1-2 sentences)"
}}"""
        )
    ]
 
    parts.append(types.Part.from_text("=== Candidate A ==="))
    for p in a_paths:
        parts.append(load_image(p))
 
    parts.append(types.Part.from_text("=== Candidate B ==="))
    for p in b_paths:
        parts.append(load_image(p))
 
    resp = client.models.generate_content(
        model=JUDGE_MODEL,
        contents=parts,
        config=types.GenerateContentConfig(
            response_mime_type="application/json",
            temperature=0.3,
        ),
    )
    return json.loads(resp.text)

This introduces a new trap: position bias. Models tend to favor whichever candidate came first (or last) regardless of content.

The fix is cheap. Ask again with the order flipped, and only count a win when both orderings agree.

def compare_debiased(a_paths, b_paths, app_name) -> str:
    """Compare twice with swapped order to cancel position bias.
 
    Returns "A" / "B" / "tie". Disagreement between the two runs counts as a tie.
    """
    first = pairwise_compare(a_paths, b_paths, app_name)["winner"]
 
    # Swap the slots, then translate the answer back into A/B terms
    swapped = pairwise_compare(b_paths, a_paths, app_name)["winner"]
    second = {"A": "B", "B": "A", "tie": "tie"}[swapped]
 
    if first == second:
        return first
    return "tie"

Across 30 pairs, 18% disagreed when I flipped the order. That 18% is exactly the set of verdicts I would have accepted at face value had I asked only once. The flip side is more encouraging: in 82% of pairs, the same candidate won no matter which slot it sat in. Absolute scoring never gave me that kind of stability.

Turn win rates into a ranking

The rest is round robin.

def rank_candidates(candidates: dict[str, list[str]], app_name: str) -> list[dict]:
    """Compare every candidate against every other, then rank by win rate.
 
    candidates: {"candidate name": [screenshot paths, ...]}
    """
    names = list(candidates.keys())
    wins = {n: 0.0 for n in names}
    matches = {n: 0 for n in names}
    log = []
 
    for a, b in combinations(names, 2):
        winner = compare_debiased(candidates[a], candidates[b], app_name)
        matches[a] += 1
        matches[b] += 1
 
        if winner == "A":
            wins[a] += 1.0
        elif winner == "B":
            wins[b] += 1.0
        else:  # tie
            wins[a] += 0.5
            wins[b] += 0.5
 
        log.append({"a": a, "b": b, "winner": winner})
        print(f"  {a} vs {b} -> {winner}")
 
    ranked = [
        {
            "name": n,
            "win_rate": round(wins[n] / matches[n], 3) if matches[n] else 0.0,
            "wins": wins[n],
            "matches": matches[n],
        }
        for n in names
    ]
    ranked.sort(key=lambda x: x["win_rate"], reverse=True)
    return ranked

Five candidates means ten pairs, twenty requests with the swap. Here is what came out.

Look at the right-hand column. Last place scores 79, first place scores 81 — two points. Same images, same model, and only the question changed, yet the win rates spread from 0.125 to 0.875, a seven-fold gap. The information was in the model the whole time. My question was what failed to extract it.

Does round robin stay affordable?

Pair count grows quadratically, so estimate before you run.

Gemini bills an image at a flat 258 tokens if it fits within 768×768, and tiles it beyond that with tokens per tile. App Store screenshots are tall (1290×2796 and similar), so sending them untouched inflates tile counts.

I downscale to 828px wide first. ASO judgment needs overall composition and headline legibility, not pixel fidelity.

Flash handled the comparisons fine. I only escalate the final pair — the top two — to Pro, to get a carefully argued rationale. Running everything on Pro costs more than the accuracy difference is worth.

There is a shortcut when the field grows. Instead of round robin, fix your current live set as a permanent opponent and run each candidate against it once — a challenger format. Five candidates becomes 4 pairs × 2 = 8 requests. You lose the full ordering, but when the only question is "did anything beat what's already shipping," that's enough. I switch to it once I have more than six candidates.

The limits I have not resolved

What this measures is which set Gemini prefers, not which set users download. I have not verified the correlation between win rate and actual conversion rate. Doing so means accumulating enough samples through App Store Connect's product page optimization, and at indie scale that takes months.

So let me be precise about what this is: not a replacement for A/B testing, but the filter that decides what goes into the A/B test. Product page optimization lets you run three treatments at once. When you have seven candidates, something has to cut it to three — and that step used to be my eye and my mood. Now it's a procedure I can rerun and inspect. That's all it is. It was enough to be worth building.

One more thing. Swapping the market changes the verdicts. Running the same five candidates with market="Japan" versus market="United States" reordered my top two. Google's own Southeast Asia report notes roughly 70% of Gemini app prompts in the region are in local languages — a reminder that markets are not interchangeable defaults. Leaving the market parameter off doesn't make the judgment neutral; it silently picks one for you. Don't leave it off.

Running this tomorrow

1. Cut to five candidates or fewer (round robin's practical ceiling)
2. Run measure_noise on identical input — ten calls, a few minutes
3. Build degraded variants and compute the discrimination margin. Under 2.0, absolute scoring can't see differences at your granularity
4. Run rank_candidates for win rates. Do not skip the order swap
5. Send the top two to App Store Connect product page optimization
6. When real data arrives, record whether the win rate predicted it

Keep doing step six long enough and you can finally answer whether Gemini's win rate correlates with real conversion. I'm not there yet. When I get there, I'll publish the number.

When I first saw that 82, I nearly believed it. Numbers reassure us whether or not they've earned it. Build a grader, then grade the grader — it looks like a detour, and it's the shortest path there is.

Thanks for reading. I hope it saves you a few weeks of trusting the wrong number.