Testing Gemini Vision for Wallpaper Auto-Classification — Real Accuracy Numbers and Pitfalls

Running a wallpaper app at scale means dealing with a problem that sounds trivial until you're doing it for the thousandth time: categorizing images. Since I started building apps independently in 2014, Beautiful HD Wallpapers (iOS and Android, 50M+ downloads) has accumulated thousands of wallpapers — each needing to be filed into one of 30+ categories like nature, architecture, abstract, animals, and cityscape.

At 3–5 seconds per image, a batch of 500 means hours of work. So I decided to test Gemini Vision for automatic classification. The first version hit 67% accuracy. Here's what went wrong, and how I got it to 87%.

First Implementation — Simple and Underperforming

My initial code was straightforward:

import google.generativeai as genai
import base64
from pathlib import Path
 
genai.configure(api_key="YOUR_GEMINI_API_KEY")
model = genai.GenerativeModel("gemini-2.0-flash")
 
def classify_wallpaper(image_path: str) -> str:
    """Classify a wallpaper image into a category (v1 - before improvements)"""
    image_data = Path(image_path).read_bytes()
    image_b64 = base64.b64encode(image_data).decode()
    
    response = model.generate_content([
        {
            "inline_data": {
                "mime_type": "image/jpeg",
                "data": image_b64
            }
        },
        "Choose one category for this wallpaper: nature, architecture, abstract, animals, cityscape, space, food, sports, other"
    ])
    return response.text.strip()
 
result = classify_wallpaper("sample_sunset.jpg")
print(result)  # → "nature"

In quick tests, it looked fine. After running 500 images through it, accuracy was 67% — roughly one in three images misclassified.

What Caused the 67% Accuracy

Problem 1: Inconsistent output format

Instead of returning "nature", the model would sometimes return "Nature", "natural scenery", or "nature (forest)". Since my downstream code compared exact strings, these all counted as wrong.

Problem 2: No rule for edge cases

A sunset cityscape — is that "nature" or "cityscape"? The model had no consistent rule, so it varied each call. The prompt gave no guidance on tiebreaking.

Problem 3: "Other" acted as a catch-all escape

With "other" in the list, the model leaned on it too frequently for borderline cases. It needs to be the last resort, not a default.

Improved Implementation — JSON Output + Structured Prompt → 87% Accuracy

Two changes pushed accuracy to 87%:

import google.generativeai as genai
import json
import base64
from pathlib import Path
 
genai.configure(api_key="YOUR_GEMINI_API_KEY")
model = genai.GenerativeModel("gemini-2.0-flash")
 
CATEGORIES = [
    "nature",       # forests, mountains, ocean, sky, plants
    "architecture", # buildings, bridges, structures
    "abstract",     # geometric, textures, patterns, CGI art
    "animals",      # animals, insects, fish
    "cityscape",    # city skylines, night scenes, streets
    "space",        # universe, stars, galaxies, planets
    "minimal",      # minimal composition with lots of whitespace
    "other",        # only when nothing above fits clearly
]
 
SYSTEM_PROMPT = """
You are a wallpaper image classifier. Follow these rules strictly:
 
1. Choose exactly one category from the provided list.
2. Return JSON in this format: { "category": "...", "confidence": 0-100, "reason": "under 20 words" }
3. For borderline cases (e.g., sunset over a city), prioritize by which element covers more visual area.
4. Use "other" only when no other category is a clear fit.
 
Category definitions:
- nature: Natural elements (forest, mountain, ocean, sky, plants, rivers) are the main subject
- architecture: Buildings, bridges, or man-made structures are the main subject
- abstract: Geometric patterns, textures, graphic patterns, CGI artwork
- animals: Animals, insects, fish, or other creatures as the main subject
- cityscape: City buildings, night scenes, streets, urban environment
- space: Universe, stars, galaxies, planets
- minimal: Simple composition with significant negative space
- other: Doesn't clearly fit any of the above
"""
 
def classify_wallpaper_v2(image_path: str) -> dict:
    """Classify a wallpaper image into a category (improved version)"""
    image_data = Path(image_path).read_bytes()
    image_b64 = base64.b64encode(image_data).decode()
    
    response = model.generate_content(
        [
            {
                "inline_data": {
                    "mime_type": "image/jpeg",
                    "data": image_b64
                }
            },
            f"Classify this image. Categories: {', '.join(CATEGORIES)}\n{SYSTEM_PROMPT}"
        ],
        generation_config=genai.types.GenerationConfig(
            response_mime_type="application/json",  # Force JSON output
            temperature=0.1,  # Reduce randomness for consistency
        )
    )
    
    result = json.loads(response.text)
    
    # Validate category name
    if result.get("category") not in CATEGORIES:
        result["category"] = "other"
        result["confidence"] = 0
    
    return result
 
# Example
result = classify_wallpaper_v2("sunset_city.jpg")
print(result)
# → {"category": "cityscape", "confidence": 72, "reason": "city buildings dominate, sunset is background"}

Setting response_mime_type="application/json" eliminated all output format inconsistencies. Setting temperature=0.1 gave consistent results across repeated calls on the same image.

Batch Processing and Rate Limit Management

Real-world wallpaper apps need to process hundreds or thousands of images at once. Gemini Flash's free tier is limited to 15 requests per minute, so batch processing needs some care:

import time
import json
from pathlib import Path
 
def batch_classify_wallpapers(
    image_dir: str,
    output_json: str,
    requests_per_minute: int = 12,  # Buffer below the 15/min limit
) -> dict:
    """
    Classify all wallpapers in a directory.
    Skips already-classified images and supports resuming mid-batch.
    """
    image_dir = Path(image_dir)
    output_path = Path(output_json)
    
    # Load existing results to support resume
    results = {}
    if output_path.exists():
        with open(output_path) as f:
            results = json.load(f)
    
    images = list(image_dir.glob("*.jpg")) + list(image_dir.glob("*.png"))
    interval = 60.0 / requests_per_minute
    
    for i, image_path in enumerate(images):
        filename = image_path.name
        
        if filename in results:
            continue  # Skip already classified
        
        try:
            result = classify_wallpaper_v2(str(image_path))
            results[filename] = {
                "category": result["category"],
                "confidence": result.get("confidence", 0),
                "reason": result.get("reason", ""),
            }
            
            # Save periodically (every 10 images)
            if (i + 1) % 10 == 0:
                with open(output_path, "w") as f:
                    json.dump(results, f, ensure_ascii=False, indent=2)
                print(f"Progress: {i+1}/{len(images)}")
            
        except Exception as e:
            print(f"Error: {filename} → {e}")
            results[filename] = {"category": "error", "confidence": 0, "reason": str(e)[:50]}
        
        time.sleep(interval)
    
    # Final save
    with open(output_path, "w") as f:
        json.dump(results, f, ensure_ascii=False, indent=2)
    
    return results

Designing for resumability saved significant time when the batch stopped mid-way due to a rate limit error.

Is 87% Accuracy Actually Good Enough?

Honestly — full automation is unrealistic. 87% paired with human review is the practical approach.

In practice, images with low confidence scores were far more likely to be wrong. Among images with confidence below 70, about 45% were misclassified. Among those above 90, accuracy was 97%.

def needs_human_review(result: dict) -> bool:
    """Determine whether a human should review this classification"""
    if result["confidence"] < 70:
        return True
    if result["category"] == "other":
        return True
    return False

With this threshold, only about 15% of images needed human review. For 500 images, that's 75 — versus reviewing all 500 manually. The workload dropped to roughly one-sixth.

What I Actually Learned from This

After integrating Gemini Vision into Beautiful HD Wallpapers' image management pipeline, I noticed something interesting: the images the AI struggled to classify were almost always the same ones I'd hesitate over myself. A sunset over a city. A lone tree that's also an architectural silhouette. The boundary cases are genuinely ambiguous.

My grandfather was a temple carpenter — he had this phrase: "working with your hands is its own kind of devotion." I came into this experiment with a slight skepticism about handing off judgment to a model. But the natural separation that emerged — AI handles clear cases fast, humans handle ambiguous ones — didn't feel like a compromise. It felt right.

The goal isn't full automation. It's freeing up time for the decisions that actually benefit from human judgment. For an indie developer running a 50M+ download app with limited hours in the day, that distinction matters more than raw accuracy numbers.

Start with 10 images, validate the prompt and temperature settings, then scale to your full batch.