Apple Vision Framework × Gemini API: Hybrid Image Recognition — Cutting Wallpaper App Cloud Inference Costs by 70%

End-to-end architecture

The runtime flow is:

Step 1. User uploads a wallpaper → UIImage arrives via Photos Step 2. Convert to CIImage, run the on-device prefilter pipeline in sequence Step 3. Rejected images get a lightweight label and a row in local SQLite — they never touch the network Step 4. Approved images are resized and JPEG-compressed (224×224 to 512×512) and posted to Gemini Vision Step 5. The Gemini response, validated against a JSON schema, is cached locally and reflected in the UI

I have been calling the on-device check the "gate" in design docs, with the mental image of a torii at the entrance to a Shinto shrine. You compose yourself at the gate before stepping into the precinct. The metaphor was useful when I had to explain the design to a collaborator.

On-device prefilter implementation

The core of the on-device side, in Swift 6 against the iOS 18 SDK:

import Vision
import CoreImage
 
struct ImagePrefilterResult {
    let shouldSendToCloud: Bool
    let coarseCategory: CoarseCategory
    let sharpnessScore: Double
    let saliencyArea: Double
    let reasonIfRejected: String?
}
 
enum CoarseCategory: String, Codable {
    case photoLike, illustrationLike, abstract, unknown
}
 
final class ImagePrefilter {
    private let saliencyRequest = VNGenerateAttentionBasedSaliencyImageRequest()
    private let classifyRequest = VNClassifyImageRequest()
 
    func analyze(ciImage: CIImage) async throws -> ImagePrefilterResult {
        let handler = VNImageRequestHandler(ciImage: ciImage, options: [:])
        try handler.perform([saliencyRequest, classifyRequest])
 
        let sharpness = sharpnessScore(of: ciImage)
        let saliencyArea = saliencyAreaRatio(from: saliencyRequest.results?.first)
        let coarse = coarseCategory(from: classifyRequest.results ?? [])
 
        if sharpness < 0.18 {
            return .init(shouldSendToCloud: false, coarseCategory: coarse,
                         sharpnessScore: sharpness, saliencyArea: saliencyArea,
                         reasonIfRejected: "too_blurry")
        }
        if saliencyArea < 0.06 {
            return .init(shouldSendToCloud: false, coarseCategory: coarse,
                         sharpnessScore: sharpness, saliencyArea: saliencyArea,
                         reasonIfRejected: "low_saliency")
        }
        if isOwnAppGeneratedWatermark(ciImage: ciImage) {
            return .init(shouldSendToCloud: false, coarseCategory: coarse,
                         sharpnessScore: sharpness, saliencyArea: saliencyArea,
                         reasonIfRejected: "own_watermark")
        }
        return .init(shouldSendToCloud: true, coarseCategory: coarse,
                     sharpnessScore: sharpness, saliencyArea: saliencyArea,
                     reasonIfRejected: nil)
    }
 
    private func sharpnessScore(of image: CIImage) -> Double {
        // Laplacian-variance approximation via CIConvolution3X3
        let kernel: [CGFloat] = [0, -1, 0, -1, 4, -1, 0, -1, 0]
        let filter = CIFilter.convolution3X3()
        filter.inputImage = image
        filter.weights = CIVector(values: kernel.map { Float($0) }, count: 9)
        filter.bias = 0.5
        guard let output = filter.outputImage else { return 0 }
        let stats = variance(of: output)
        return min(stats / 0.42, 1.0)
    }
}

The 0.18 sharpness threshold is calibrated against 2,400 images that human reviewers flagged as "honestly cannot tell what this is" plus 3,800 clean images from the same app. The ROC curve gave AUC 0.93, and I picked the operating point at that knee. A different app — say one that accepts very dark or very minimal photos by design — would need its own calibration here.

The 0.06 saliency-area threshold is set so that genuine abstract wallpapers (which carry attention across the whole frame) are not killed off accidentally. The choice was: never throw away a real candidate, even at the cost of letting a few junk frames through.

Sending to Gemini Vision and structuring the response

Only gate-approved images reach the cloud. For cost, every image is resized to a maximum side of 512 px and re-encoded as JPEG at quality 0.78 before posting.

import GoogleGenerativeAI
 
struct WallpaperClassification: Codable {
    let primaryCategory: String
    let secondaryCategory: String?
    let mood: String
    let recommendedTags: [String]
    let dominantColors: [String]
    let suitableScreenTypes: [String]
}
 
final class WallpaperClassifier {
    private let model: GenerativeModel
 
    init(apiKey: String) {
        self.model = GenerativeModel(
            name: "gemini-2.5-flash",
            apiKey: apiKey,
            generationConfig: GenerationConfig(
                temperature: 0.2,
                topP: 0.95,
                maxOutputTokens: 512,
                responseMIMEType: "application/json",
                responseSchema: Self.responseSchema
            )
        )
    }
 
    func classify(image: UIImage,
                  coarseHint: CoarseCategory) async throws -> WallpaperClassification {
        let resized = image.resizedKeepingAspect(maxSide: 512)
        let jpegData = resized.jpegData(compressionQuality: 0.78)!
 
        let prompt = """
        You are the classifier of a wallpaper app. Look at this one image and
        return classification as JSON.
 
        An on-device coarse classifier already labeled it as "\(coarseHint.rawValue)".
        Treat that purely as a hint, not as ground truth.
 
        primaryCategory must be one of:
        nature, abstract, person, animal, vehicle, architecture, space, urban,
        anime, japanese_art, cyberpunk, lofi, minimalist, geometric, food, sports,
        seasonal, other
 
        mood must be one of:
        calm, energetic, mysterious, romantic, melancholic, playful, sacred, neutral
 
        recommendedTags: 3 to 6 lowercase snake_case items.
        dominantColors: up to 3 #rrggbb values.
        suitableScreenTypes: any subset of "iphone_lock", "iphone_home", "ipad", "mac_desktop".
        """
 
        let response = try await model.generateContent(
            prompt,
            ModelContent.Part.data(mimetype: "image/jpeg", jpegData)
        )
        guard let text = response.text,
              let data = text.data(using: .utf8) else {
            throw ClassifierError.emptyResponse
        }
        return try JSONDecoder().decode(WallpaperClassification.self, from: data)
    }
}

Three details worth pointing out.

First, responseMIMEType: "application/json" plus responseSchema together give a strong contract on the output. With prompt-only JSON instructions, malformed JSON happened on 12% of responses in my data. With schema binding it dropped to 0.4%.

Second, the coarse hint from on-device is passed as context, but explicitly labeled as a hint rather than ground truth. With "treat that purely as a hint" included, edge-case agreement with human labels went up about 4 points. Without that line, Gemini tended to over-anchor on the coarse label.

Third, temperature: 0.2 is intentional. For classification you want stable, reproducible outputs, not creative variance. Low temperature also tends to use slightly fewer tokens, which compounds with everything else above.

What the production numbers look like

Thirty days after switching to the hybrid setup, the real numbers were:

Total uploads: 543,800 Images filtered out by the on-device gate: 290,200 (53.4%) Images sent to Gemini Vision: 253,600 Monthly Gemini Vision cost: roughly 173 USD (the equivalent month under the all-cloud setup was estimated at about 600 USD) Cost reduction: 71.2% Average upload-to-category latency: 920 ms → 410 ms Support tickets about "my upload didn't get categorized": 7 across the month (0.0024%)

A breakdown of why images were rejected:

Low sharpness (too_blurry): 198,400 Low saliency (low_saliency): 71,200 Detected as our own watermark (own_watermark): 20,600

The watermark case in particular was a pleasant surprise. I had not anticipated that 600–800 of my own previously-generated wallpapers would be re-uploaded into the app every day, and skipping those alone covers a meaningful portion of the savings.

Gotcha 1: Saliency unfairly rejects abstract art

In the first week, multiple users wrote in that "my abstract wallpapers no longer get a category." Investigation showed that VNGenerateAttentionBasedSaliencyImageRequest returns near-zero scores for uniform gradients and geometric abstract pieces — exactly the kind of frames where there is no single object to focus on.

The fix was to combine the attention-based saliency with the object-based variant (VNGenerateObjectnessBasedSaliencyImageRequest) and only reject when both fall below threshold. As a second safety net, if VNClassifyImageRequest returned any of "abstract," "pattern," or "texture" in its top labels, I skipped the saliency check entirely.

False rejection rate moved from 0.13% to 0.018% after that. The on-device gate should prioritize "never throw away a correct answer" over raw precision; the user-visible quality of the overall pipeline depends on it.

Gotcha 2: Safety settings drop legitimate cultural imagery

After a few weeks, I noticed BLOCK responses creeping up on photos of Shinto shrine grounds, jizō statues, and altars. With Gemini Vision's default safety_settings, religious symbolism can trip MEDIUM-level safety filters and get blocked outright.

For an app that explicitly celebrates Japanese traditional motifs and contemplative imagery, this is unacceptable. I left HARM_CATEGORY_HATE_SPEECH and the other high-risk categories on their defaults, but loosened the religion-relevant categories from MEDIUM to BLOCK_ONLY_HIGH and added logging so I could continue to monitor what is being flagged.

If your domain touches religious or culturally specific content, tuning safety_settings deliberately is part of the job. The point is not to "turn safety off" — it is to take responsibility for the threshold that suits your domain.

Gotcha 3: Battery drain and thermal throttling

Apple Vision Framework runs on the Neural Engine, but a tight loop over 1,000+ images in a row still triggers thermal throttling, and users will (politely) notice. My wallpaper app has a background batch-categorize mode, and that mode was where the warmth showed up.

What I ended up with was a three-stage governor:

1. Monitor ProcessInfo.thermalState and inject a 5-second pause whenever it reaches .serious or higher
2. If battery drops below 20%, abort the local batch and queue Gemini Vision sends behind a network-availability gate
3. Only allow full-scale background processing when the device is both charging and on Wi-Fi, and preferably at night

For a personal-developer wallpaper app, this kind of restrained operation is essentially a prerequisite for running on-device AI without harming experience.

Cross-porting to Android (Google ML Kit)

The Android app uses the same hybrid structure: Google ML Kit's ImageLabeler and SubjectSegmenter for the on-device prefilter, and the same Gemini Vision API in the cloud, with a thin server layer that normalizes any iOS/Android client drift.

Two differences stood out on Android.

The first is device variance. On a Pixel-class device ML Kit responds essentially instantly, but on lower-end Android hardware the same operations can run 5×–10× slower. I added an 800 ms timeout in front of the threshold check: if the local request misses that deadline, the image bypasses the gate and gets sent to Gemini Vision anyway. Failing open is the safer default here.

The second is revenue context. iOS and Android AdMob eCPM are not equal — in my wallpaper app, Android eCPM runs at roughly 60% of iOS. The same Gemini Vision cost therefore eats more of the per-image revenue on Android. I tightened the Android-side gate to keep the cloud-send rate at 41%. Cost optimization on cloud inference only makes sense when you design it together with the revenue side.

Where to start: the smallest first step

Going fully hybrid in one shot makes debugging painful. The order I actually used, as a solo developer:

In week one I shipped only the measurement step. Every image still went to Gemini Vision for classification, but every image also had its sharpness, saliency, and size logged locally. A small Swift Charts dashboard then projected, retroactively, "if I had been gating at threshold X, this is how many calls I would have skipped, and these are the categories I might have rejected by mistake." This made it possible to reason about the savings and the risk in numbers, before any code path actually changed.

In week two I switched the gate on, but with very conservative thresholds (around the 5th percentile of the observed distribution). I measured false-reject rate from real user support tickets.

In week three, once that rate was inside the budget I had set for myself (around 0.01%), I started tightening the thresholds in small steps.

"Don't reach for the final design first — first make the savings visible." If you are an indie developer trying to optimize cloud inference, that order is what I would recommend.

Reflections

Hybrid inference is not a design that avoids Gemini. It is a design that makes room for Gemini to keep being useful — by being deliberate about which images it sees.

Both of my grandfathers were temple carpenters, and the idea of "discipline that lets something last" still shapes how I think about building. For AI to last in production, I suspect similar discipline is becoming necessary.

An all-cloud inference path is far simpler to write. Without cost pressure, I would probably have stayed there. But as an indie developer running this as a real business, having a structure where I can decide on my own when to cut and when to send leaves me less exposed to swings in model pricing or model behavior over time.

The next experiment I am working on is distilling some of Gemini Vision's frequent patterns back into the on-device side, so that the most common classifications can happen entirely offline. I will write that up separately when I have real numbers. Thanks for reading.