If you run a wallpaper app, one chore becomes unavoidable: sorting thousands of images into categories like "night scene," "minimal," or "animals."
Every time a few hundred new assets arrived, I used to look at each one and tag it by hand. That worked at first, but as the library grew it stopped being something a human should be doing. Handing image classification to Gemini's multimodal API came out of that very practical need.
Then, recently, I went to run that classification script after leaving it alone for a while — and stopped short. The package it depended on had become the previous generation. It wasn't broken, exactly. But if I were writing the same thing today, the shape would be completely different.
So I rebuilt the whole thing while I was migrating. What follows is the result: image input, resolution tuning, structured output, retries, and cost tracking as one connected implementation.
The classification script I'd left on the old SDK
This script originally used @google/generative-ai. The current package for the Gemini API is @google/genai, and the call shape has changed.
Three differences stand out. The client is created with new GoogleGenAI({ apiKey }). Generation collapses into a single ai.models.generateContent({ model, contents, config }), with the model name and settings both living inside the argument. And the response text is now a property, res.text, rather than a function call, res.text().
That last one quietly ate my afternoon. Leave res.text() in place and it fails with "text is not a function." The error message is perfectly accurate — but my head still held the old SDK's shape, so I spent a while suspecting entirely the wrong code.
import { GoogleGenAI } from "@google/genai";
import fs from "node:fs";
const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
async function analyzeImage(imagePath, prompt) {
const base64Image = fs.readFileSync(imagePath).toString("base64");
const res = await ai.models.generateContent({
model: "gemini-flash-latest",
contents: [
{
role: "user",
parts: [
{ inlineData: { mimeType: "image/jpeg", data: base64Image } },
{ text: prompt },
],
},
],
});
// A property now, not the old SDK's res.text()
return res.text;
}Images go into inlineData with a mimeType and a Base64 string, lined up inside parts. Being able to mix text and images in the same parts array is the honest appeal of multimodal, and the order carries meaning. I put the image first and the instruction second, so the judgment happens right after the instruction is read.
The model name gemini-flash-latest currently points to Gemini 3.5 Flash. It's worth pausing here. An alias silently re-points whenever a newer model ships — convenient, but awkward for batch work.
If the same image comes back in a different category than it did last month, consistency with the thousands already classified quietly breaks. For accumulating work like classification, I pin the model to a specific version and switch on my own schedule. For one-off summaries or draft generation, I follow latest. Same API, opposite call, decided by the nature of the job.
I've written up this idea of confining "which model you depend on" to a single place in Never scrambling again when a deprecation notice arrives. Now that this migration is my third, I keenly wish I'd drawn that layer from the start.
Resolution sets both accuracy and cost
This was the single change that mattered most when I switched from manual work to automation. Wallpaper originals are often 4K, but sending them as-is for classification is wasteful. Gemini splits images into tiles for token counting, so higher resolution means a steeply higher cost per image.
For a classification task, shrinking the long edge to around 1024px barely dented accuracy. If anything, losing the extra detail made big-picture calls like "night versus day" more stable. Showing less and getting more right feels backwards — though on reflection, people work the same way. Stand further back and the overall impression is easier to read.
import sharp from "sharp";
// Lighten for classification: 1024px long edge, JPEG quality 80 is plenty
async function toClassificationJpeg(inputPath) {
return sharp(inputPath)
.resize(1024, 1024, { fit: "inside", withoutEnlargement: true })
.jpeg({ quality: 80 })
.toBuffer();
}Just inserting the resize step roughly halved the bill for processing the same number of images. For an app funded by ad revenue (AdMob), that gap turns directly into breathing room. Cutting cost in half while holding accuracy steady was a more realistic win, as a solo developer, than squeezing out one extra point of precision.
withoutEnlargement: true is unglamorous but worth keeping. Some assets are only around 800px on the long edge, and stretching those to 1024px adds no information — only tokens.
Dropping the regex that stripped the JSON
This was the least dignified part of the old implementation. The prompt asked politely for "JSON only," and the returned string had its JSON code fence stripped by a regular expression before JSON.parse.
Asking politely means being occasionally refused. On days when a preamble sentence showed up, parsing failed and the whole run halted. Stopping mid-way through thousands of images because of one of them is not good for the nerves.
Now responseSchema declares the type and the model is held to the structure.
import { GoogleGenAI, Type } from "@google/genai";
const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
const classificationSchema = {
type: Type.OBJECT,
properties: {
category: {
type: Type.STRING,
// Closing this with enum stops invented categories
enum: ["night", "minimal", "animal", "nature", "abstract"],
description: "Pick exactly one category that best fits the image",
},
confidence: {
type: Type.NUMBER,
description: "0.0-1.0. If uncertain, return an honestly low value",
},
},
required: ["category", "confidence"],
propertyOrdering: ["category", "confidence"],
};
async function classifyWallpaper(inputPath) {
const buffer = await toClassificationJpeg(inputPath);
const res = await ai.models.generateContent({
model: "gemini-3.5-flash",
contents: [
{
role: "user",
parts: [
{ inlineData: { mimeType: "image/jpeg", data: buffer.toString("base64") } },
{ text: "Classify this wallpaper image." },
],
},
],
config: {
responseMimeType: "application/json",
responseSchema: classificationSchema,
},
});
// No fence stripping, no worrying about preambles
return JSON.parse(res.text);
}enum is what did the work. Back when categories came back as free strings, "night scene" and "night" and "nighttime" mixed together and needed cleaning up. Once the list was closed, values outside it simply stopped arriving, and I deleted the downstream normalization code entirely.
Using description as a place for the rule on the ground, rather than a description of the type, paid off later too. "0.0-1.0" alone produces less honest spread than adding "if uncertain, return an honestly low value." When everything comes back at 0.95, the threshold in the next section stops meaning anything.
That said, structured output guarantees shape, not meaning. I've separated out where that line falls in Trusting Gemini's structured output in production.
Gate on confidence
Not making the classifier fully automatic was another lesson from running it. Always have Gemini return a confidence score, and only review the low ones by hand. That one step stopped misclassifications from going public unnoticed.
async function classifyBatch(paths, reviewThreshold = 0.7) {
const auto = [];
const needsReview = [];
for (const path of paths) {
const { category, confidence } = await classifyWallpaper(path);
const record = { path, category, confidence };
if (confidence >= reviewThreshold) {
auto.push(record);
} else {
needsReview.push(record); // collect only the uncertain ones
}
}
return { auto, needsReview };
}Out of thousands of images, only about one in ten needed a human look. The other 90% were auto-accepted above 0.7 confidence and flowed into the publish pipeline without review.
There's no deep reasoning behind 0.7. I started at 0.5, watched what landed in the review box, and nudged it up until it settled around here. Pushing to 0.8 nearly doubled the review pile without catching meaningfully more errors. The habit of actually looking inside the review box matters more than the number itself.
"Send only the uncertain ones back to a person," rather than "let the AI do everything." That line is what lets quality and speed coexist.
Retry only on 429 and 5xx
Run a real batch and you will hit transient errors. If you retry everything, you also hammer genuine request mistakes (400) forever. Limiting retries to rate limits (429) and temporary server failures (500/503) is the practical choice.
async function generateWithRetry(params, maxRetries = 3) {
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
const res = await ai.models.generateContent(params);
return res.text;
} catch (error) {
const status = error?.status ?? error?.response?.status;
// Bad input never succeeds on resend. Throw immediately.
if (status === 400) throw error;
// Back off only for rate limits and transient failures
if (status === 429 || status === 500 || status === 503) {
const wait = 1000 * 2 ** (attempt - 1);
await new Promise((r) => setTimeout(r, wait));
continue;
}
throw error;
}
}
throw new Error(`Did not complete after ${maxRetries} attempts`);
}After throwing 400 straight back, the logs became far easier to read. Where they had been buried in "waiting to retry," the input mistakes that actually needed fixing finally stood out.
One more thing: the backoff intervals can stay crude. One second, two, four. A solo developer's batch runs overnight, so a few seconds bother nobody. I started building variable jitter here and stopped myself partway. Knowing where not to spend effort is part of running things alone.
Track the running total, not the estimate
Last, a humble one that paid off: cumulative cost tracking. The per-image price is tiny, but run thousands of images several times a month and it adds up. The response's usageMetadata carries the tokens actually consumed, so summing that — rather than an estimate — is the accurate way.
async function analyzeAndCount(params, counter) {
const res = await ai.models.generateContent(params);
const usage = res.usageMetadata;
counter.totalTokens += usage?.totalTokenCount ?? 0;
counter.images += 1;
return { text: res.text, tokens: usage?.totalTokenCount };
}Carrying counter.images alongside gives you average tokens per image, which turned out to be the most honest yardstick for whether a resolution change did anything. Watching that average drop visibly after moving to a 1024px long edge is what finally made me trust the resize step.
I reset this total at the start of each month and read it next to AdMob revenue. Whether the API cost of automated classification is worth it against the hours saved and the ad income — that single comparison is what decides if a feature stays or gets retired.
What to try next
Start with ten images, declare category and confidence in a responseSchema, and have gemini-flash-latest fill them in. Feel what it's like to skip writing fence-stripping code once, and you won't go back.
From there, insert the 1024px resize and watch the average token count move. Set the threshold at 0.5 and look at what collects in the review box. Progress in that order and the shape naturally drifts toward something that survives higher volume.
If you have a script still sleeping on the old SDK, rebuilding it while you migrate is probably the fastest route. That's how it went for me. I hope it helps if you are sitting on a large pile of assets too.