When JSON Shapes Drift Through Gemini's OpenAI Compatibility Layer: Three Steps to response_json_schema Without Rewriting Callers

I was rewriting a small piece of my wallpaper app one evening — the part that reads the caption on a submitted image and picks exactly one tag. The code had started life on the OpenAI SDK, and as an indie developer I had simply swapped the base_url and model name to point it at Gemini. What came back looked like JSON, but now and then it arrived wrapped in a sentence of explanation, and json.loads stopped right there.

My first move was to add "return JSON only" to the prompt. Then I added it again. It didn't help. The problem wasn't the prompt at all: no schema was being sent in the first place.

If you've tried the OpenAI-compatible endpoint and felt that structured output "just doesn't work," this is what I'd check first, and how I moved to the Gemini SDK's response_json_schema without throwing the existing code away.

First, confirm the schema is actually being sent

Google's "OpenAI compatibility" page in the Gemini API docs includes an example of structured output through the compatibility layer. At the same time, the Google AI Developers Forum has several threads reporting that structured output doesn't take effect through that layer, or that propertyOrdering isn't respected.

When I looked at my own code, I was passing only response_format={"type": "json_object"}. That says "reply in JSON." It says nothing about which JSON. Key names and allowed values were left to the prose of the prompt, and the shape drifted whenever the model tried to be helpful.

The check is simple. Match what you pass in response_format against this table.

If you're on the second row, moving to the third row alone fixes a lot. Only if the shape still drifts after that is it worth questioning the route itself.

Step 1: Stay on the compatibility layer, send a schema, and validate

Don't switch SDKs yet. Define a Pydantic model, send its JSON Schema as json_schema, and validate the returned string against the same model.

import logging
import os
from typing import Literal, Optional
 
from openai import OpenAI
from pydantic import BaseModel, ValidationError
 
# Check the Models page on ai.google.dev before relying on a model name
MODEL = "gemini-4-hydrogen"
SYSTEM = "From the wallpaper caption, choose the single closest tag."
 
 
class WallpaperTag(BaseModel):
    label: Literal["landscape", "portrait", "abstract", "other"]
    reason: str
 
 
compat = OpenAI(
    api_key=os.environ["GEMINI_API_KEY"],
    base_url="https://generativelanguage.googleapis.com/v1beta/openai/",
)
 
 
def classify_compat(caption: str) -> Optional[WallpaperTag]:
    completion = compat.chat.completions.create(
        model=MODEL,
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": caption},
        ],
        response_format={
            "type": "json_schema",
            "json_schema": {
                "name": "wallpaper_tag",
                "schema": WallpaperTag.model_json_schema(),
            },
        },
    )
    text = completion.choices[0].message.content or ""
    try:
        return WallpaperTag.model_validate_json(text)
    except ValidationError as e:
        # Keep malformed replies so you can count which route produced them
        logging.warning("compat schema mismatch: %s | raw=%r", e.errors()[0]["type"], text[:200])
        return None

The part that matters most is logging failed validations instead of swallowing them. Once you can count failures, the decision about the next step comes from a record rather than a hunch.

Step 2: Keep the function boundary, swap the inside to the Gemini SDK

If failures remain after Step 1, or you want optional fields and field order to follow Gemini's own rules, move the call to the Gemini SDK (google-genai). To avoid touching any callers, keep the function name, arguments, and return type identical to Step 1.

from typing import Optional
 
from google import genai
from google.genai import types
from pydantic import ValidationError
 
# Reuses MODEL, SYSTEM and WallpaperTag from above
native = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
 
 
def classify_native(caption: str) -> Optional[WallpaperTag]:
    res = native.models.generate_content(
        model=MODEL,
        contents=caption,
        config=types.GenerateContentConfig(
            system_instruction=SYSTEM,
            response_mime_type="application/json",
            # The same JSON Schema, built from the same Pydantic model
            response_json_schema=WallpaperTag.model_json_schema(),
        ),
    )
    try:
        return WallpaperTag.model_validate_json(res.text or "")
    except ValidationError as e:
        logging.warning("native schema mismatch: %s | raw=%r", e.errors()[0]["type"], (res.text or "")[:200])
        return None

The schema comes from WallpaperTag.model_json_schema() in one place so I never write the "sent" shape and the "validated" shape twice. The route can change; the definition of correct stays in a single Pydantic model.

One note as of November 2026: in the official docs, the generateContent Structured outputs page carries a "Legacy" label, and the Interactions API has its own structured output page. I stayed on generateContent here to keep the diff small against existing code, but for something written from scratch, I'd read the Interactions API page first.

Step 3: Switch with an environment variable and run both for a while

Flipping everything at once is how you end up with a knot in your stomach when you want to roll back. I made the route selectable with one environment variable, compared both on my own captions, and only then changed the default.

import os
from typing import Optional
 
 
def classify(caption: str) -> Optional[WallpaperTag]:
    backend = os.environ.get("TAG_BACKEND", "compat")
    if backend == "native":
        return classify_native(caption)
    return classify_compat(caption)
 
 
def shadow_compare(captions: list[str]) -> None:
    """Send the same input through both routes; count label mismatches and failures."""
    diff = fail_compat = fail_native = 0
    for c in captions:
        a, b = classify_compat(c), classify_native(c)
        fail_compat += a is None
        fail_native += b is None
        if a and b and a.label != b.label:
            diff += 1
            print(f"DIFF {a.label} / {b.label} : {c[:60]}")
    print(f"total={len(captions)} mismatches={diff} compat_failures={fail_compat} sdk_failures={fail_native}")

shadow_compare never goes into the production path; it's a tool you run once locally. In my case, the mismatches clustered on captions like "city lights at night" that could reasonably be landscape or abstract. That pointed less at the route and more at how loosely I had defined my own allowed values.

Decide what "correct" means in one place before you change the route. That line did more for this rewrite than anything else.

Where people tend to trip

• Believing json_object sends a schema. It only sets the type. Send json_schema together with the schema body.
• Reading Pydantic's Optional as "the key may be missing." Optional allows null; a missing key is a separate case. I cover that distinction in designing structured output that survives schema changes.
• Assuming an enum guarantees allowed values. A consistent shape doesn't guarantee meaning. The fixes in when responseSchema enum returns unexpected strings apply to the compatibility route too.
• Validating an empty res.text and crashing. Some replies carry no text, for example after a safety filter. Passing an empty string into validation, as above, sends it to the log where you can trace it.

If parsing itself is what fails, fixing structured output JSON errors may help as well.

Closing

The compatibility layer is a generous entry point: it lets you try Gemini without rewriting much. Whether to leave production running through it, though, is a decision you can make after looking at the validation log.

If you take one thing from this, run the Step 1 code with json_schema on the compatibility layer and keep one day of validation failures. That count is the most honest input I know for deciding whether to switch SDKs.