Judging Gemma 4 and Nemotron 3 Nano Omni on 100 of My Own Images, Not a Benchmark Score

Not "100 representative images" — 100 stratified ones

The obvious first move is to pull 100 images at random from the full set. That move is a trap.

My wallpaper distribution is lopsided: the top three categories (minimal, night scenes, nature) account for about 58% of everything. Sample 100 at random and those three claim 58 slots, leaving 42 to split across the remaining 27 categories. Rare categories land one or two images each. They can be completely broken and the aggregate barely twitches.

I don't want the average. I want the places where it breaks. So I stratify.

Three images minimum per category — 90 across 30 categories. The remaining 10 are a separate boundary-case pool: images where a user once told me the classification was wrong, or where I hesitated myself. That pool is, in my view, the real eval set.

# build_eval_set.py — build the eval set by stratified sampling
import json
import random
from collections import defaultdict
from pathlib import Path
 
random.seed(42)  # always pin this — reproducibility is the whole point
 
CATALOG = Path("data/wallpapers.jsonl")   # {"id":..., "path":..., "label":...}
BOUNDARY = Path("data/boundary_ids.txt")  # images previously flagged or ambiguous
OUT = Path("data/eval_set.jsonl")
 
PER_CATEGORY = 3
BOUNDARY_QUOTA = 10
 
 
def load_catalog(path: Path) -> list[dict]:
    return [json.loads(line) for line in path.read_text().splitlines() if line.strip()]
 
 
def stratified_sample(rows: list[dict], per_category: int) -> list[dict]:
    buckets: dict[str, list[dict]] = defaultdict(list)
    for row in rows:
        buckets[row["label"]].append(row)
 
    picked: list[dict] = []
    for label, items in sorted(buckets.items()):
        if len(items) < per_category:
            # thin categories are exactly the fragile ones — surface them, don't silently shrink
            print(f"[warn] {label}: only {len(items)} images (< {per_category})")
        picked.extend(random.sample(items, min(per_category, len(items))))
    return picked
 
 
def main() -> None:
    rows = load_catalog(CATALOG)
    boundary_ids = set(BOUNDARY.read_text().split())
 
    base = [r for r in rows if r["id"] not in boundary_ids]
    sampled = stratified_sample(base, PER_CATEGORY)
 
    boundary_rows = [r for r in rows if r["id"] in boundary_ids]
    sampled += random.sample(boundary_rows, min(BOUNDARY_QUOTA, len(boundary_rows)))
 
    with OUT.open("w") as f:
        for row in sampled:
            f.write(json.dumps(row, ensure_ascii=False) + "\n")
 
    print(f"eval set: {len(sampled)} images / {len(set(r['label'] for r in sampled))} categories")
 
 
if __name__ == "__main__":
    main()

random.seed(42) is pinned so that when you swap models you compare the same 100 images. Let the eval set drift and you will never separate a model difference from a sampling difference. I measured for several days without pinning it, noticed, and threw the results away.

The warning for thin categories earns its place too. If a category has fewer than three images, the shortage is in your data, not the model — swapping models will not fix it. Knowing that before you evaluate changes the order of what you do next.

Accuracy hides where you went wrong

With 100 images ready, run them through both the local model and the cloud, then look at accuracy and the confusion matrix together.

# evaluate.py — collect predictions, report accuracy and confusions
import json
from collections import Counter
from pathlib import Path
from typing import Callable
 
EVAL_SET = Path("data/eval_set.jsonl")
 
 
def evaluate(predict: Callable[[str], str], name: str) -> dict:
    rows = [json.loads(line) for line in EVAL_SET.read_text().splitlines()]
 
    confusion: dict[tuple[str, str], int] = Counter()
    per_label_total: dict[str, int] = Counter()
    hits = 0
 
    for row in rows:
        truth = row["label"]
        pred = predict(row["path"])
        confusion[(truth, pred)] += 1
        per_label_total[truth] += 1
        if pred == truth:
            hits += 1
 
    accuracy = hits / len(rows)
 
    # rank the errors by count — this is the part worth reading
    errors = sorted(
        ((t, p, n) for (t, p), n in confusion.items() if t != p),
        key=lambda x: x[2],
        reverse=True,
    )
 
    print(f"\n=== {name} ===")
    print(f"accuracy: {accuracy:.2%} ({hits}/{len(rows)})")
    print("top confusions:")
    for truth, pred, n in errors[:5]:
        share = n / per_label_total[truth]
        print(f"  {truth} -> {pred}: {n} ({share:.0%} of {truth})")
 
    return {"name": name, "accuracy": accuracy, "confusion": dict(confusion)}

Here is what my 100 images produced.

Read 82% on its own and it sounds tolerable. That was my first reaction to that row.

Open the confusion matrix and the picture changes. Of Gemma 4's 18 errors, seven were night scene ⇄ cityscape swaps in both directions. Those, honestly, don't matter. A photo of a city at night is fine in either list, and nobody browsing the app is inconvenienced.

The rest did matter. Two images labeled animal came back as person. Small in count. Visible as a defect.

Errors don't all cost the same

Accuracy counts every error as one. In shipped products, I have never once had all errors be equal.

So write down a cost matrix: for each way of confusing true label t with prediction p, how much does it hurt? Nothing more sophisticated than that.

Apply the weights and divide again.

# cost_weighted_error.py — error rate weighted by what each mistake costs
import json
from pathlib import Path
 
# {(true, pred): cost}. Anything unlisted falls back to the default
COST_MATRIX_PATH = Path("config/cost_matrix.json")
DEFAULT_COST = 1.0
 
 
def load_cost_matrix() -> dict[tuple[str, str], float]:
    raw = json.loads(COST_MATRIX_PATH.read_text())
    return {(item["true"], item["pred"]): item["cost"] for item in raw}
 
 
def cost_weighted_error(confusion: dict, costs: dict) -> float:
    """Takes {(true, pred): count} and returns the weighted error rate."""
    total = sum(confusion.values())
    if total == 0:
        return 0.0
 
    penalty = 0.0
    for (truth, pred), n in confusion.items():
        if truth == pred:
            continue
        penalty += costs.get((truth, pred), DEFAULT_COST) * n
 
    return penalty / total
 
 
def report(result: dict) -> None:
    costs = load_cost_matrix()
    # evaluate.py's return value has string keys once serialized — restore the tuples
    confusion = {tuple(k.split("\t")): v for k, v in result["confusion"].items()}
    cwe = cost_weighted_error(confusion, costs)
 
    print(f"{result['name']}: accuracy={result['accuracy']:.2%} / cost-weighted error={cwe:.3f}")

Same 100 images, re-scored.

Accuracy separated 82% and 79% by three points. Cost-weighted error separates 0.31 and 0.44 — a different animal entirely. Nemotron's mistakes clustered in the across-families band. My reading is that a single still image mapped to a single label never crosses a modality boundary, so the thing Nemotron is built to be good at simply never comes up.

That table is where Nemotron dropped out of this use case. Not a bad model. A bad placement.

Adopt on the budget, not on "≥90% accuracy"

Now the adoption question is answerable.

You see rules like "adopt if accuracy is 90% or higher" everywhere. Where does the 90 come from? I wrote one of those rules myself, couldn't defend the number when asked, and replaced it.

What you want is a quality budget:

1. Count how many cost-1.0 errors (the across-families kind) your current cloud setup actually produces. For me, across a full 38,000-image re-run, two to three a month escalated into user complaints
2. Decide, in your own words, where that stops being acceptable. I set mine at "more than five a month and the app starts feeling careless"
3. Convert your eval set's cost-weighted error into that unit

Step three is the whole trick. If 0.12 on 100 images corresponds to two or three complaints a month, then 0.31 is roughly 2.6× — call it five to eight a month. That crosses the line I just drew.

Full replacement: rejected. Run 82% through the conversion and it lands outside the budget.

But stopping there would have wasted the measurement. Back in the confusion matrix, Gemma 4's precision on the top three categories was 97% — that part was genuinely trustworthy. Accept the local answer only when it lands in one of those three categories with high confidence, and send everything else to the cloud. That shape brings a full re-run from about ¥3,400 down to roughly ¥1,430, and from five hours to about three and a half.

So the answer was neither adopt nor reject. It was partial adoption — and I could not have drawn that line without the 100 images.

How to implement that split and keep the threshold honest are separate problems. The routing implementation and its cost design live in Cutting a ¥32,000 Gemini API bill to ¥9,000 with Gemma 4 local inference — designing the hybrid router; extracting the confidence score itself is covered in Measuring classification confidence with Gemini API logprobs; and the way a hand-set threshold quietly ages is the subject of Recalibrating Flash-vs-Pro routing thresholds with shadow evaluation. This article stops one step earlier, at the question of whether the model belongs in your stack at all.

Re-quantize, and the conclusion quietly rots

A few weeks after deciding, I moved Gemma 4 from Q4_K_M to Q5_K_M. That is a change in the direction of better quality, so I expected the check to be a formality.

Same 100 images:

Accuracy improved by two points. Cost-weighted error got worse.

The breakdown explained it. The majority categories picked up correct answers, while the errors that remained in the rare categories shifted from adjacent-category swaps to across-family swaps. Fewer mistakes, worse mistakes.

Watching accuracy alone, that change ships as an improvement. And once shipped, tracing a rise in complaints back to a quantization bump would have been close to impossible.

So the check runs automatically, every time.

# regression_check.py — fail CI on the delta, not the headline
import json
import sys
from pathlib import Path
 
BASELINE = Path("data/baseline.json")   # frozen at adoption time
CURRENT = Path("data/current.json")     # this run's evaluation
 
# even if accuracy rose, refuse a weighted-error regression beyond this margin
CWE_TOLERANCE = 0.03
 
 
def main() -> int:
    base = json.loads(BASELINE.read_text())
    now = json.loads(CURRENT.read_text())
 
    delta_acc = now["accuracy"] - base["accuracy"]
    delta_cwe = now["cost_weighted_error"] - base["cost_weighted_error"]
 
    print(f"accuracy: {base['accuracy']:.2%} -> {now['accuracy']:.2%} ({delta_acc:+.2%})")
    print(f"cost-weighted error: {base['cost_weighted_error']:.3f} -> "
          f"{now['cost_weighted_error']:.3f} ({delta_cwe:+.3f})")
 
    if delta_cwe > CWE_TOLERANCE:
        print(f"\nFAIL: weighted error regressed past the {CWE_TOLERANCE} tolerance. "
              f"Rejecting despite the accuracy gain.")
        return 1
 
    print("\nPASS")
    return 0
 
 
if __name__ == "__main__":
    sys.exit(main())

Twenty-five lines. Running a local model means taking back the maintenance Google had been quietly doing for you. This much scaffolding won't remove that work, but it does remove the part where you fail to notice.

On edge hardware, the GPU is not what gives out first

Evaluation runs into hardware sooner than you'd like, and this part only shows up in operation.

First, storage I/O binds before compute does. Every image load is a large read, so the GPU sits idle waiting on the SSD. Stream 4K wallpapers back to back and this is unmistakable.

Second, heat. Edge enclosures have limited cooling, and sustained inference throttles throughput by around 30%. On my hardware, p95 latency stretched from 1.8s to 3.1s in the back half of a 30-minute run.

That feeds straight back into evaluation design. The speed you measure on ten images from a cold start is not the speed of a 38,000-image run. The five-hour and three-and-a-half-hour figures above are post-correction numbers. Extrapolating cold-start throughput by 38,000 would have made my estimate about 20% too optimistic.

Third, the update flow. On a cloud API, Google handles the upgrades. Locally, tracking quantization builds and catching regressions is your job — which is the previous section.

Measure on the back half of a 30-minute run, not on ten cold images. That lesson is cheap to learn this way and expensive to learn the other way.

Where to start

If you are weighing up local multimodal, build the 100 images before you collect a single comparison table.

Three per category, stratified. Ten previously flagged images in a separate pool. Pin the seed and save it. Half a day, usually less.

The cost matrix can wait. Even just reading the confusion matrix and sorting the mistakes into "this one hurts" and "this one is fine" in your own words draws a line that a headline score never showed you.

I was one step away from adopting on the strength of 82%. The number wasn't wrong. I just hadn't checked what it represented about my own work. Working as an indie developer means owning that judgment alone, which is exactly why a small instrument like this earns its keep.