Testing Gemini Vision for Wallpaper Auto-Classification — Real Accuracy Numbers and Pitfalls

Running a wallpaper app means dealing with a problem that sounds trivial until you're doing it for the thousandth time: categorizing images. The app I maintain as a solo developer has accumulated thousands of wallpapers — each needing to be filed into one of 30+ categories like nature, architecture, abstract, animals, and cityscape.

At 3–5 seconds per image, a batch of 500 means hours of work. So I decided to test Gemini Vision for automatic classification. The first version hit 67% accuracy. Here's what went wrong, and how I got it to 87%.

First Implementation — Simple and Underperforming

My initial code was straightforward:

import google.generativeai as genai
import base64
from pathlib import Path
 
genai.configure(api_key="YOUR_GEMINI_API_KEY")
model = genai.GenerativeModel("gemini-2.0-flash")
 
def classify_wallpaper(image_path: str) -> str:
    """Classify a wallpaper image into a category (v1 - before improvements)"""
    image_data = Path(image_path).read_bytes()
    image_b64 = base64.b64encode(image_data).decode()
    
    response = model.generate_content([
        {
            "inline_data": {
                "mime_type": "image/jpeg",
                "data": image_b64
            }
        },
        "Choose one category for this wallpaper: nature, architecture, abstract, animals, cityscape, space, food, sports, other"
    ])
    return response.text.strip()
 
result = classify_wallpaper("sample_sunset.jpg")
print(result)  # → "nature"

In quick tests, it looked fine. After running 500 images through it, accuracy was 67% — roughly one in three images misclassified.

What Caused the 67% Accuracy

Problem 1: Inconsistent output format

Instead of returning "nature", the model would sometimes return "Nature", "natural scenery", or "nature (forest)". Since my downstream code compared exact strings, these all counted as wrong.

Problem 2: No rule for edge cases

A sunset cityscape — is that "nature" or "cityscape"? The model had no consistent rule, so it varied each call. The prompt gave no guidance on tiebreaking.

Problem 3: "Other" acted as a catch-all escape

With "other" in the list, the model leaned on it too frequently for borderline cases. It needs to be the last resort, not a default.

Improved Implementation — JSON Output + Structured Prompt → 87% Accuracy

Two changes pushed accuracy to 87%:

import google.generativeai as genai
import json
import base64
from pathlib import Path
 
genai.configure(api_key="YOUR_GEMINI_API_KEY")
model = genai.GenerativeModel("gemini-2.0-flash")
 
CATEGORIES = [
    "nature",       # forests, mountains, ocean, sky, plants
    "architecture", # buildings, bridges, structures
    "abstract",     # geometric, textures, patterns, CGI art
    "animals",      # animals, insects, fish
    "cityscape",    # city skylines, night scenes, streets
    "space",        # universe, stars, galaxies, planets
    "minimal",      # minimal composition with lots of whitespace
    "other",        # only when nothing above fits clearly
]
 
SYSTEM_PROMPT = """
You are a wallpaper image classifier. Follow these rules strictly:
 
1. Choose exactly one category from the provided list.
2. Return JSON in this format: { "category": "...", "confidence": 0-100, "reason": "under 20 words" }
3. For borderline cases (e.g., sunset over a city), prioritize by which element covers more visual area.
4. Use "other" only when no other category is a clear fit.
 
Category definitions:
- nature: Natural elements (forest, mountain, ocean, sky, plants, rivers) are the main subject
- architecture: Buildings, bridges, or man-made structures are the main subject
- abstract: Geometric patterns, textures, graphic patterns, CGI artwork
- animals: Animals, insects, fish, or other creatures as the main subject
- cityscape: City buildings, night scenes, streets, urban environment
- space: Universe, stars, galaxies, planets
- minimal: Simple composition with significant negative space
- other: Doesn't clearly fit any of the above
"""
 
def classify_wallpaper_v2(image_path: str) -> dict:
    """Classify a wallpaper image into a category (improved version)"""
    image_data = Path(image_path).read_bytes()
    image_b64 = base64.b64encode(image_data).decode()
    
    response = model.generate_content(
        [
            {
                "inline_data": {
                    "mime_type": "image/jpeg",
                    "data": image_b64
                }
            },
            f"Classify this image. Categories: {', '.join(CATEGORIES)}\n{SYSTEM_PROMPT}"
        ],
        generation_config=genai.types.GenerationConfig(
            response_mime_type="application/json",  # Force JSON output
            temperature=0.1,  # Reduce randomness for consistency
        )
    )
    
    result = json.loads(response.text)
    
    # Validate category name
    if result.get("category") not in CATEGORIES:
        result["category"] = "other"
        result["confidence"] = 0
    
    return result
 
# Example
result = classify_wallpaper_v2("sunset_city.jpg")
print(result)
# → {"category": "cityscape", "confidence": 72, "reason": "city buildings dominate, sunset is background"}

Setting response_mime_type="application/json" eliminated all output format inconsistencies. Setting temperature=0.1 gave consistent results across repeated calls on the same image.

Batch Processing and Rate Limit Management

Real-world wallpaper apps need to process hundreds or thousands of images at once. Gemini Flash's free tier is limited to 15 requests per minute, so batch processing needs some care:

import time
import json
from pathlib import Path
 
def batch_classify_wallpapers(
    image_dir: str,
    output_json: str,
    requests_per_minute: int = 12,  # Buffer below the 15/min limit
) -> dict:
    """
    Classify all wallpapers in a directory.
    Skips already-classified images and supports resuming mid-batch.
    """
    image_dir = Path(image_dir)
    output_path = Path(output_json)
    
    # Load existing results to support resume
    results = {}
    if output_path.exists():
        with open(output_path) as f:
            results = json.load(f)
    
    images = list(image_dir.glob("*.jpg")) + list(image_dir.glob("*.png"))
    interval = 60.0 / requests_per_minute
    
    for i, image_path in enumerate(images):
        filename = image_path.name
        
        if filename in results:
            continue  # Skip already classified
        
        try:
            result = classify_wallpaper_v2(str(image_path))
            results[filename] = {
                "category": result["category"],
                "confidence": result.get("confidence", 0),
                "reason": result.get("reason", ""),
            }
            
            # Save periodically (every 10 images)
            if (i + 1) % 10 == 0:
                with open(output_path, "w") as f:
                    json.dump(results, f, ensure_ascii=False, indent=2)
                print(f"Progress: {i+1}/{len(images)}")
            
        except Exception as e:
            print(f"Error: {filename} → {e}")
            results[filename] = {"category": "error", "confidence": 0, "reason": str(e)[:50]}
        
        time.sleep(interval)
    
    # Final save
    with open(output_path, "w") as f:
        json.dump(results, f, ensure_ascii=False, indent=2)
    
    return results

Designing for resumability saved significant time when the batch stopped mid-way due to a rate limit error.

How I Actually Picked the Confidence Threshold

The "review anything below 70" rule wasn't a guess. I hand-labeled the same 500 images, bucketed the model's confidence scores in steps of ten, and measured accuracy per bucket before settling on a number.

import json
from collections import defaultdict
 
def confidence_report(results_json: str, truth_json: str) -> None:
    """Bucket confidence scores by ten and print accuracy per bucket."""
    results = json.load(open(results_json))
    truth = json.load(open(truth_json))  # {filename: correct_category}
 
    buckets = defaultdict(lambda: {"total": 0, "correct": 0})
    for filename, r in results.items():
        if filename not in truth or r["category"] == "error":
            continue
        b = min(int(r["confidence"]) // 10 * 10, 90)
        buckets[b]["total"] += 1
        if r["category"] == truth[filename]:
            buckets[b]["correct"] += 1
 
    running = 0
    for b in sorted(buckets):
        total, correct = buckets[b]["total"], buckets[b]["correct"]
        acc = correct / total * 100 if total else 0
        running += total
        print(f"{b:>3}-{b+9:<3} n={total:>3}  acc={acc:5.1f}%  cumulative={running:>3}")
 
confidence_report("classified.json", "ground_truth.json")

Here is what the 500-image run produced.

Confidence	Images	Accuracy	Notes
0–59	21	38.1%	Assume a human must look
60–69	52	59.6%	Roughly half are wrong
70–79	88	81.8%	Practical lower bound
80–89	147	92.5%	Safe to auto-accept
90–100	192	97.4%	Effectively settled

Accuracy jumps 22 points between the 60s and the 70s. That gap is where the threshold belongs. Dropping to 60 shrinks the review pile but ships images that are wrong about 40% of the time. Raising it to 80 pushes the review pile to 32% of the batch, which starts to defeat the purpose. Seventy came out of the measurements as the point where accuracy and effort balanced best — not out of intuition.

This isn't a one-time calculation. Rewriting category definitions or switching model versions shifts the distribution. I re-run the same 500 images after every prompt change and re-check where the jump lands. It costs an afternoon, but it lets me widen the auto-accept range with far more confidence than leaving a stale threshold in place.

Is 87% Accuracy Actually Good Enough?

Honestly — full automation is unrealistic. 87% paired with human review is the practical approach.

In practice, images with low confidence scores were far more likely to be wrong. Among images with confidence below 70, about 45% were misclassified. Among those above 90, accuracy was 97%.

def needs_human_review(result: dict) -> bool:
    """Determine whether a human should review this classification"""
    if result["confidence"] < 70:
        return True
    if result["category"] == "other":
        return True
    return False

With this threshold, only about 15% of images needed human review. For 500 images, that's 75 — versus reviewing all 500 manually. The workload dropped to roughly one-sixth.

What I Actually Learned from This

After integrating Gemini Vision into my image management pipeline, I noticed something interesting: the images the AI struggled to classify were almost always the same ones I'd hesitate over myself. A sunset over a city. A lone tree that's also an architectural silhouette. The boundary cases are genuinely ambiguous.

I came into this experiment with a slight skepticism about handing off judgment to a model. But the separation that emerged — the model handles clear cases fast, I handle the ambiguous ones — didn't feel like a compromise. It felt right.

The goal isn't full automation. It's freeing up time for the decisions that actually benefit from human judgment. When you're a solo developer with limited hours in the day, that distinction matters more than raw accuracy numbers.

Start with 10 images, validate the prompt and temperature settings, then scale to your full batch.