Judging Gemma 4 and Nemotron 3 Nano Omni on 100 of My Own Images, Not a Benchmark Score

Not "100 representative images" — 100 stratified ones

The obvious first move is to pull 100 images at random from the full set. That move is a trap.

My wallpaper distribution is lopsided: the top three categories (minimal, night scenes, nature) account for about 58% of everything. Sample 100 at random and those three claim 58 slots, leaving 42 to split across the remaining 27 categories. Rare categories land one or two images each. They can be completely broken and the aggregate barely twitches.

I don't want the average. I want the places where it breaks. So I stratify.

Three images minimum per category — 90 across 30 categories. The remaining 10 are a separate boundary-case pool: images where a user once told me the classification was wrong, or where I hesitated myself. That pool is, in my view, the real eval set.

# build_eval_set.py — build the eval set by stratified sampling
import json
import random
from collections import defaultdict
from pathlib import Path
 
random.seed(42)  # always pin this — reproducibility is the whole point
 
CATALOG = Path("data/wallpapers.jsonl")   # {"id":..., "path":..., "label":...}
BOUNDARY = Path("data/boundary_ids.txt")  # images previously flagged or ambiguous
OUT = Path("data/eval_set.jsonl")
 
PER_CATEGORY = 3
BOUNDARY_QUOTA = 10
 
 
def load_catalog(path: Path) -> list[dict]:
    return [json.loads(line) for line in path.read_text().splitlines() if line.strip()]
 
 
def stratified_sample(rows: list[dict], per_category: int) -> list[dict]:
    buckets: dict[str, list[dict]] = defaultdict(list)
    for row in rows:
        buckets[row["label"]].append(row)
 
    picked: list[dict] = []
    for label, items in sorted(buckets.items()):
        if len(items) < per_category:
            # thin categories are exactly the fragile ones — surface them, don't silently shrink
            print(f"[warn] {label}: only {len(items)} images (< {per_category})")
        picked.extend(random.sample(items, min(per_category, len(items))))
    return picked
 
 
def main() -> None:
    rows = load_catalog(CATALOG)
    boundary_ids = set(BOUNDARY.read_text().split())
 
    base = [r for r in rows if r["id"] not in boundary_ids]
    sampled = stratified_sample(base, PER_CATEGORY)
 
    boundary_rows = [r for r in rows if r["id"] in boundary_ids]
    sampled += random.sample(boundary_rows, min(BOUNDARY_QUOTA, len(boundary_rows)))
 
    with OUT.open("w") as f:
        for row in sampled:
            f.write(json.dumps(row, ensure_ascii=False) + "\n")
 
    print(f"eval set: {len(sampled)} images / {len(set(r['label'] for r in sampled))} categories")
 
 
if __name__ == "__main__":
    main()

random.seed(42) is pinned so that when you swap models you compare the same 100 images. Let the eval set drift and you will never separate a model difference from a sampling difference. I measured for several days without pinning it, noticed, and threw the results away.

The warning for thin categories earns its place too. If a category has fewer than three images, the shortage is in your data, not the model — swapping models will not fix it. Knowing that before you evaluate changes the order of what you do next.

Line up the predictors before you measure anything

The predict callable that evaluate.py takes is the seam where models get swapped. Standardise it first, or the prompt and the post-processing drift a little with every comparison. I let that happen once and could no longer say whether a gap came from the model or from my own wording.

Two conditions are enough. The prompt must be byte-identical across models. Whatever comes back must land on one of the thirty categories.

# predictors.py — one seam, one prompt, one output contract
import base64
import json
import os
import re
from pathlib import Path
 
import requests
 
LABELS = json.loads(Path("config/labels.json").read_text())  # canonical 30 names
PROMPT = (
    "Classify the image into exactly one of the categories below.\n"
    "Output the category name only — no explanation, no punctuation.\n\n"
    + "\n".join(f"- {label}" for label in LABELS)
)
 
 
def _encode(path: str) -> str:
    return base64.b64encode(Path(path).read_bytes()).decode()
 
 
def _normalize(raw: str) -> str:
    """Map raw output onto a canonical label, or return unknown."""
    text = re.sub(r"[\s.,:;\-*`]", "", raw).lower()
    for label in LABELS:
        if label.lower() in text:
            return label
    return "unknown"  # never snap to the nearest label
 
 
def local_predict(path: str, model: str = "gemma-4-mm") -> str:
    """For OpenAI-compatible endpoints such as llama.cpp server or Ollama."""
    res = requests.post(
        "http://127.0.0.1:8080/v1/chat/completions",
        json={
            "model": model,
            "temperature": 0,
            "max_tokens": 16,
            "messages": [{
                "role": "user",
                "content": [
                    {"type": "text", "text": PROMPT},
                    {"type": "image_url",
                     "image_url": {"url": f"data:image/jpeg;base64,{_encode(path)}"}},
                ],
            }],
        },
        timeout=120,
    )
    res.raise_for_status()
    return _normalize(res.json()["choices"][0]["message"]["content"])
 
 
def cloud_predict(path: str, model: str = "gemini-3.5-flash") -> str:
    res = requests.post(
        f"https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent",
        headers={"x-goog-api-key": os.environ["GEMINI_API_KEY"]},
        json={
            "contents": [{
                "parts": [
                    {"text": PROMPT},
                    {"inline_data": {"mime_type": "image/jpeg", "data": _encode(path)}},
                ]
            }],
            "generationConfig": {"temperature": 0, "maxOutputTokens": 16},
        },
        timeout=60,
    )
    res.raise_for_status()
    return _normalize(res.json()["candidates"][0]["content"]["parts"][0]["text"])

The unknown return is deliberate. You could snap the output to the nearest label by edit distance, but doing that rewards the model that ignored your instructions. Whether a model can follow an output contract is part of what you are deciding on, so quietly rescuing it deletes the evidence.

Because unknown never matches the truth label, it counts as an error on its own. If rows like (night, unknown) start stacking up in the confusion matrix, that is a format-compliance problem rather than a vision problem — and now you can tell the two apart.

In my run, the unknown rate was 0% for Gemma 4 and 3% for Nemotron 3 Nano Omni. Nemotron still liked to open with something like "This is a photograph of a city at night", and capping max_tokens at 16 did not always stop the preamble. Those three images are inside the 79% quoted above.

One more thing: this predict is not evaluation-only. Production classification calls the same function. Keep a separate implementation for evaluation and one of them goes stale the next time you touch the prompt.

Accuracy hides where you went wrong

With 100 images ready, run them through both the local model and the cloud, then look at accuracy and the confusion matrix together.

# evaluate.py — collect predictions, report accuracy and confusions
import json
from collections import Counter
from pathlib import Path
from typing import Callable
 
EVAL_SET = Path("data/eval_set.jsonl")
 
 
def evaluate(predict: Callable[[str], str], name: str) -> dict:
    rows = [json.loads(line) for line in EVAL_SET.read_text().splitlines()]
 
    confusion: dict[tuple[str, str], int] = Counter()
    per_label_total: dict[str, int] = Counter()
    hits = 0
 
    for row in rows:
        truth = row["label"]
        pred = predict(row["path"])
        confusion[(truth, pred)] += 1
        per_label_total[truth] += 1
        if pred == truth:
            hits += 1
 
    accuracy = hits / len(rows)
 
    # rank the errors by count — this is the part worth reading
    errors = sorted(
        ((t, p, n) for (t, p), n in confusion.items() if t != p),
        key=lambda x: x[2],
        reverse=True,
    )
 
    print(f"\n=== {name} ===")
    print(f"accuracy: {accuracy:.2%} ({hits}/{len(rows)})")
    print("top confusions:")
    for truth, pred, n in errors[:5]:
        share = n / per_label_total[truth]
        print(f"  {truth} -> {pred}: {n} ({share:.0%} of {truth})")
 
    # Tuple keys will not survive JSON, so flatten them here
    return {
        "name": name,
        "accuracy": accuracy,
        "confusion": {f"{t}\t{p}": n for (t, p), n in confusion.items()},
    }

Here is what my 100 images produced.

Read 82% on its own and it sounds tolerable. That was my first reaction to that row.

Open the confusion matrix and the picture changes. Of Gemma 4's 18 errors, seven were night scene ⇄ cityscape swaps in both directions. Those, honestly, don't matter. A photo of a city at night is fine in either list, and nobody browsing the app is inconvenienced.

The rest did matter. Two images labeled animal came back as person. Small in count. Visible as a defect.

Errors don't all cost the same

Accuracy counts every error as one. In shipped products, I have never once had all errors be equal.

So write down a cost matrix: for each way of confusing true label t with prediction p, how much does it hurt? Nothing more sophisticated than that.

Apply the weights and divide again.

# cost_weighted_error.py — error rate weighted by what each mistake costs
import json
from pathlib import Path
 
# {(true, pred): cost}. Anything unlisted falls back to the default
COST_MATRIX_PATH = Path("config/cost_matrix.json")
DEFAULT_COST = 1.0
 
 
def load_cost_matrix() -> dict[tuple[str, str], float]:
    raw = json.loads(COST_MATRIX_PATH.read_text())
    return {(item["true"], item["pred"]): item["cost"] for item in raw}
 
 
def cost_weighted_error(confusion: dict, costs: dict) -> float:
    """Takes {(true, pred): count} and returns the weighted error rate."""
    total = sum(confusion.values())
    if total == 0:
        return 0.0
 
    penalty = 0.0
    for (truth, pred), n in confusion.items():
        if truth == pred:
            continue
        penalty += costs.get((truth, pred), DEFAULT_COST) * n
 
    return penalty / total
 
 
def report(result: dict) -> None:
    costs = load_cost_matrix()
    # evaluate.py's return value has string keys once serialized — restore the tuples
    confusion = {tuple(k.split("\t")): v for k, v in result["confusion"].items()}
    cwe = cost_weighted_error(confusion, costs)
 
    print(f"{result['name']}: accuracy={result['accuracy']:.2%} / cost-weighted error={cwe:.3f}")

Same 100 images, re-scored.

Accuracy separated 82% and 79% by three points. Cost-weighted error separates 0.31 and 0.44 — a different animal entirely. Nemotron's mistakes clustered in the across-families band. My reading is that a single still image mapped to a single label never crosses a modality boundary, so the thing Nemotron is built to be good at simply never comes up.

That table is where Nemotron dropped out of this use case. Not a bad model. A bad placement.

Adopt on the budget, not on "≥90% accuracy"

Now the adoption question is answerable.

You see rules like "adopt if accuracy is 90% or higher" everywhere. Where does the 90 come from? I wrote one of those rules myself, couldn't defend the number when asked, and replaced it.

What you want is a quality budget:

1. Count how many cost-1.0 errors (the across-families kind) your current cloud setup actually produces. For me, across a full 38,000-image re-run, two to three a month escalated into user complaints
2. Decide, in your own words, where that stops being acceptable. I set mine at "more than five a month and the app starts feeling careless"
3. Convert your eval set's cost-weighted error into that unit

Step three is the whole trick. If 0.12 on 100 images corresponds to two or three complaints a month, then 0.31 is roughly 2.6× — call it five to eight a month. That crosses the line I just drew.

Full replacement: rejected. Run 82% through the conversion and it lands outside the budget.

But stopping there would have wasted the measurement. Back in the confusion matrix, Gemma 4's precision on the top three categories was 97% — that part was genuinely trustworthy. Accept the local answer only when it lands in one of those three categories with high confidence, and send everything else to the cloud. That shape brings a full re-run from about ¥3,400 down to roughly ¥1,430, and from five hours to about three and a half.

So the answer was neither adopt nor reject. It was partial adoption — and I could not have drawn that line without the 100 images.

How to implement that split and keep the threshold honest are separate problems. The routing implementation and its cost design live in Cutting a ¥32,000 Gemini API bill to ¥9,000 with Gemma 4 local inference — designing the hybrid router; extracting the confidence score itself is covered in Measuring classification confidence with Gemini API logprobs; and the way a hand-set threshold quietly ages is the subject of Recalibrating Flash-vs-Pro routing thresholds with shadow evaluation. This article stops one step earlier, at the question of whether the model belongs in your stack at all.

Re-quantize, and the conclusion quietly rots

A few weeks after deciding, I moved Gemma 4 from Q4_K_M to Q5_K_M. That is a change in the direction of better quality, so I expected the check to be a formality.

Same 100 images:

Accuracy improved by two points. Cost-weighted error got worse.

The breakdown explained it. The majority categories picked up correct answers, while the errors that remained in the rare categories shifted from adjacent-category swaps to across-family swaps. Fewer mistakes, worse mistakes.

Watching accuracy alone, that change ships as an improvement. And once shipped, tracing a rise in complaints back to a quantization bump would have been close to impossible.

So the check runs automatically, every time.

# regression_check.py — fail CI on the delta, not the headline
import json
import sys
from pathlib import Path
 
BASELINE = Path("data/baseline.json")   # frozen at adoption time
CURRENT = Path("data/current.json")     # this run's evaluation
 
# even if accuracy rose, refuse a weighted-error regression beyond this margin
CWE_TOLERANCE = 0.03
 
 
def main() -> int:
    base = json.loads(BASELINE.read_text())
    now = json.loads(CURRENT.read_text())
 
    delta_acc = now["accuracy"] - base["accuracy"]
    delta_cwe = now["cost_weighted_error"] - base["cost_weighted_error"]
 
    print(f"accuracy: {base['accuracy']:.2%} -> {now['accuracy']:.2%} ({delta_acc:+.2%})")
    print(f"cost-weighted error: {base['cost_weighted_error']:.3f} -> "
          f"{now['cost_weighted_error']:.3f} ({delta_cwe:+.3f})")
 
    if delta_cwe > CWE_TOLERANCE:
        print(f"\nFAIL: weighted error regressed past the {CWE_TOLERANCE} tolerance. "
              f"Rejecting despite the accuracy gain.")
        return 1
 
    print("\nPASS")
    return 0
 
 
if __name__ == "__main__":
    sys.exit(main())

Twenty-five lines. Running a local model means taking back the maintenance Google had been quietly doing for you. This much scaffolding won't remove that work, but it does remove the part where you fail to notice.

The ground moves on someone else's schedule

Re-quantizing was my own change, so the prompt to re-check came from my side. The harder case is when the baseline shifts while I do nothing at all.

The 0.12 in this article was measured against a pinned gemini-3.5-flash. The lineup has since moved on, and gemini-3.8-flash is generally available. Its introductory rate is $0.75 in / $3.75 out per million tokens — the same introductory rate the previous generation carried, held through December 31, 2026.

That "same rate" is the part that gave me pause. A generation change that doesn't move the bill removes one of the few signals you'd have noticed by accident. A price increase shows up at month end. A flat price tells you nothing about the fact that your comparison floor just changed underneath you.

And if you call an alias like gemini-flash-latest, the thing behind it rotates on Google's schedule, not yours. Convenient as an operational default. Less convenient when you are the one holding a baseline, because the reference point moves on its own.

What this needs isn't another accuracy run. It's writing down what you measured against, in the same file as the measurement. The Gemini API response carries a modelVersion field that reports the version that actually answered, even when you called an alias. Stamp that string into the baseline when you save it.

# baseline_stamp.py — record what the baseline was measured against
import json
import os
from datetime import datetime, timezone
from pathlib import Path
 
import requests
 
BASELINE = Path("data/baseline.json")
 
 
def resolve_model_version(model: str) -> str:
    """One minimal call, purely to read back which version answered"""
    res = requests.post(
        f"https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent",
        headers={"x-goog-api-key": os.environ["GEMINI_API_KEY"]},
        json={
            "contents": [{"parts": [{"text": "ping"}]}],
            "generationConfig": {"maxOutputTokens": 1},
        },
        timeout=30,
    )
    res.raise_for_status()
    # Call an alias, get back the concrete version. That's the value worth keeping
    return res.json().get("modelVersion", "unknown")
 
 
def stamp(result: dict, model: str) -> None:
    stamped = dict(result)
    stamped["model_alias"] = model
    stamped["model_version"] = resolve_model_version(model)
    stamped["measured_at"] = datetime.now(timezone.utc).isoformat()
    BASELINE.write_text(json.dumps(stamped, ensure_ascii=False, indent=2))
    print(f"baseline stamped: {stamped['model_alias']} -> {stamped['model_version']}")
 
 
def assert_same_ground(model: str) -> None:
    """Call before the regression check. If the floor moved, refuse to compare"""
    base = json.loads(BASELINE.read_text())
    recorded = base.get("model_version")
    now = resolve_model_version(model)
    if now != recorded:
        raise SystemExit(
            f"baseline was measured on {recorded}, but {model} now resolves to {now}. "
            "Re-measure on the same 100 images before comparing."
        )

Calling assert_same_ground("gemini-flash-latest") at the top of regression_check.py is enough to stop the run on the morning the floor changes. The remedy when it fires is a single thing: push the same 100 images through again and re-stamp. With a stratified eval set already on disk, that's under half a day.

My own changes stay in my memory. Google's don't — and that asymmetry is, I think, the whole argument for turning the record into a mechanism rather than a note to self.

At 100 images, how many points count as noise?

The quantization section reported 82% to 84%. Two points of improvement. Those two points mean nothing on their own.

Accuracy over 100 images carries binomial noise. A Wilson interval on 82% of 100 gives a 95% confidence interval of roughly 73.3% to 88.3% — fifteen points wide. A two-point move disappears inside it completely.

# interval.py — Wilson interval, and the sample size to shrink it
from math import sqrt
 
Z = 1.96  # 95%
 
 
def wilson(hits: int, n: int, z: float = Z) -> tuple[float, float]:
    if n == 0:
        return (0.0, 0.0)
    p = hits / n
    denom = 1 + z * z / n
    center = (p + z * z / (2 * n)) / denom
    margin = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denom
    return (center - margin, center + margin)
 
 
def required_n(p: float, half_width: float, z: float = Z) -> int:
    """Images needed for a half-width of `half_width` (0.03 = plus/minus 3 points)."""
    return int((z * z * p * (1 - p)) / (half_width * half_width)) + 1
 
 
if __name__ == "__main__":
    lo, hi = wilson(82, 100)
    print(f"82/100 -> 95% CI [{lo:.1%}, {hi:.1%}]")
    for hw in (0.05, 0.03, 0.02):
        print(f"+/-{hw:.0%} needs about {required_n(0.82, hw)} images")

Which prints:

82/100 -> 95% CI [73.3%, 88.3%]
+/-5% needs about 227 images
+/-3% needs about 631 images
+/-2% needs about 1418 images

Talking about accuracy at three-point resolution takes more than six hundred images. One hundred does not have that resolution, and pretending otherwise is where a lot of model comparisons go wrong.

That does not make 100 images useless. What I was reading in those 100 was never the accuracy figure — it was where the errors clustered. Seven in night-vs-city, two in animal-to-person: that shape is a different kind of information from a confidence interval, and it stabilises at a far smaller sample than the mean does.

The 0.31 to 0.38 move after re-quantizing gets read the same way. The size of the 0.07 matters less than the direction — errors migrating from adjacent swaps to across-family swaps. Direction survives a small sample; a two-point delta does not.

So I never chose between 82% and 79% on the strength of three points. What I decided on was the gap between 0.31 and 0.44, and what sat inside it. The accuracy table is only the doorway into that.

There is a case for adding images: when cost-weighted error lands close to the budget line. That is when I run required_n and extend the set. I don't build six hundred up front because every boundary case costs a walk back through old complaints to select it, and that cost scales linearly with the count.

On edge hardware, the GPU is not what gives out first

Evaluation runs into hardware sooner than you'd like, and this part only shows up in operation.

First, storage I/O binds before compute does. Every image load is a large read, so the GPU sits idle waiting on the SSD. Stream 4K wallpapers back to back and this is unmistakable.

Second, heat. Edge enclosures have limited cooling, and sustained inference throttles throughput by around 30%. On my hardware, p95 latency stretched from 1.8s to 3.1s in the back half of a 30-minute run.

That feeds straight back into evaluation design. The speed you measure on ten images from a cold start is not the speed of a 38,000-image run. The five-hour and three-and-a-half-hour figures above are post-correction numbers. Extrapolating cold-start throughput by 38,000 would have made my estimate about 20% too optimistic.

Third, the update flow. On a cloud API, Google handles the upgrades. Locally, tracking quantization builds and catching regressions is your job — which is the previous section.

Measure on the back half of a 30-minute run, not on ten cold images. That lesson is cheap to learn this way and expensive to learn the other way.

Where to start

If you are weighing up local multimodal, build the 100 images before you collect a single comparison table.

Three per category, stratified. Ten previously flagged images in a separate pool. Pin the seed and save it. Half a day, usually less.

The cost matrix can wait. Even just reading the confusion matrix and sorting the mistakes into "this one hurts" and "this one is fine" in your own words draws a line that a headline score never showed you.

I was one step away from adopting on the strength of 82%. The number wasn't wrong. I just hadn't checked what it represented about my own work. Working as an indie developer means owning that judgment alone, which is exactly why a small instrument like this earns its keep.