Measuring Classification Confidence with Gemini API Logprobs — A Practical Walkthrough

Picture a simple classification task: you ask Gemini to label an incoming email as "inquiry", "complaint", or "other". You get back a single word. It looks fine on the surface — but you have no idea whether the model picked it with 0.99 confidence or with 0.42, neck-and-neck with another option. The first time I shipped this kind of pipeline as an indie developer, the lack of visibility into that confidence was the single biggest source of stress.

Turning on responseLogprobs puts that confidence number in your hands. Here I'll walk through how to enable logprobs in the Gemini API and, more importantly, how to turn the raw values into an "auto-process or send to a human reviewer" gate that actually holds up in production. The flag itself is one boolean; the design decisions around it are where almost all the leverage lives, and that's what I want to spend most of the words on.

The problem logprobs solves

When Gemini answers "inquiry", was that the safe top choice — or did "complaint" come in a close second? The downstream system you trust to act on the label looks very different in those two worlds.

Logprobs are the natural-log probabilities Gemini assigned to each generated token. With them you can:

• Route only low-confidence cases to human review, like an active-learning loop
• Compare prompt A vs. prompt B by looking at confidence on the same inputs
• Detect "almost right" failures in A/B tests with a real number, not a hunch
• Auto-process whatever clears your quality gate, and only spend human time on the rest

The implementation cost is surprisingly small for how much it changes downstream design — that was my honest first impression after a few days of using it.

Enabling logprobs in the Gemini API

Here's the smallest useful setup using the google-genai Python SDK. response_logprobs=True turns logprobs on, and logprobs=N asks for the top-N alternatives at each position (1–5).

# pip install google-genai
from google import genai
from google.genai import types
import os
 
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
 
config = types.GenerateContentConfig(
    response_logprobs=True,    # return logprobs for chosen tokens
    logprobs=3,                # also return top-3 alternatives per position
    max_output_tokens=4,       # one or two tokens is plenty for classification
    temperature=0,             # determinism on 2.x; silently ignored on 3.x (see below)
)
 
resp = client.models.generate_content(
    model="gemini-2.5-flash",
    contents=(
        "Classify the following email into exactly one of "
        "{inquiry, complaint, other}.\n"
        "Email: My order hasn't arrived. Order number 12345.\n"
        "Answer (one word only):"
    ),
    config=config,
)
 
print(resp.text)
print(resp.candidates[0].logprobs_result)

Three things matter here. First, temperature=0. We want to compare confidence as "how strongly the model preferred this token", and randomness destroys that interpretation. That setting only lands on the 2.x line, though — the 3.x models handle it differently, which I cover further down. Second, cap max_output_tokens low. Most classification answers fit in one or two tokens, and longer outputs just add noise. Third, logprobs=3 is plenty — even when you ask for ten alternatives, you'll mostly look at the top two or three.

Reading the response

The logprobs_result field has two parts: chosen_candidates (the tokens actually emitted) and top_candidates (the top-N alternatives at each step). For classification, you usually only care about the first meaningful token.

import math
 
def label_confidence(resp):
    """Return the chosen label and its probability in [0, 1]."""
    chosen = resp.candidates[0].logprobs_result.chosen_candidates
    if not chosen:
        return None, 0.0
    first = chosen[0]
    prob = math.exp(first.log_probability)
    return first.token, prob
 
label, p = label_confidence(resp)
print(f"label={label!r}  confidence={p:.3f}")
# Example: label='inquiry'  confidence=0.987

log_probability is a natural log, so math.exp() gives you a probability between 0 and 1. Anything around 0.99 is essentially "the model is sure"; once you drop below 0.5, you're in territory where another label was a serious contender.

A practical gate: auto-process when confident, review when not

This is the pattern that paid for itself fastest in my own work. One threshold, one decision, and your automated share goes up or down accordingly.

def classify_with_gate(text: str, threshold: float = 0.85):
    """Auto-accept if confidence >= threshold; otherwise queue for review."""
    resp = client.models.generate_content(
        model="gemini-2.5-flash",
        contents=(
            "Classify the following email into exactly one of "
            "{inquiry, complaint, other}.\n"
            f"Email: {text}\nAnswer (one word only):"
        ),
        config=config,
    )
    label, prob = label_confidence(resp)
 
    if label is None:
        return {"status": "ERROR", "reason": "empty response"}
 
    if prob >= threshold:
        return {"status": "AUTO", "label": label, "confidence": prob}
 
    # Surface the alternatives so the review UI has something useful to show
    top = resp.candidates[0].logprobs_result.top_candidates
    alternatives = [
        (c.token, math.exp(c.log_probability))
        for c in (top[0].candidates if top else [])
    ]
    return {
        "status": "REVIEW",
        "label": label,
        "confidence": prob,
        "alternatives": alternatives,
    }

I'd recommend not picking threshold out of thin air. Run a week or two of real traffic, log every (label, confidence) pair, and look at the distribution. Once you can say "at threshold 0.85 we auto-accept 78% and humans correct 4% of those", every threshold tweak becomes a number, not a vibe.

This pairs nicely with type-safe structured output via Pydantic and the patterns in production rate-limiting for the Gemini API. Logprobs give you the gate; structured output gives you the schema; rate-limit design keeps the lights on.

Calibrating the threshold with 100 examples

"Should I just set the threshold to 0.85?" — the honest answer is that it depends on your domain. What I actually do is pull 100 historical inputs and build a small spreadsheet:

• Ground truth: the label a human eventually confirmed
• Model label: what classify_with_gate returned
• Confidence: the probability of the chosen token

Sort by confidence descending and you'll usually see something like "above 0.95 the error rate is zero", "0.85–0.94 has 2% errors", "0.70–0.84 has 9% errors". Pick the highest confidence band where the error rate is still acceptable for your use case (say, 2%) and you have a defensible threshold backed by your own data — not a number you copied from a blog post.

If no threshold gives you an acceptable error rate, the issue is almost never logprobs. It's usually that the prompt is ambiguous about where one label ends and the next begins. Tighten the label definitions, add a few canonical examples, and retest. Logprobs are a useful instrument on top of a well-defined prompt, not a fix for a fuzzy one.

When logprobs aren't reliable

Logprobs are useful, but they're not a universal "trust this number" knob. Three places I've personally been burned:

First, long generations. Logprobs are per-token, so once your output is ten or more tokens, no single number captures "confident at the start, drifting later". In practice, lean on logprobs only when the answer fits in one to a few tokens.

Second, loose prompts. If your prompt says "classify this casually", the model often emits filler words and your label slips to token two or three. End the prompt with a tight constraint like Answer (one word only): so the first chosen token actually is the label.

Third, temperature behaves differently depending on the model generation. On the 2.x line, any sampling means the chosen token can change between runs and the probability you read reflects sampling rather than the model's stable preference — so keep temperature=0 if a threshold depends on it. On the 3.x line the failure runs the other way: you can set it, and it won't land. That one gets its own section.

A fourth pitfall worth flagging: tokenizer surprises. Gemini's tokenizer doesn't always split on the boundaries you expect. A label like "complaint" might actually be one token in English but split into multiple subword tokens for Japanese labels such as "苦情". When that happens, the "first chosen token" is a fragment, not the full label. The fix is straightforward — read more than the first chosen token if your label can span subwords, or constrain output to a small set of single-token labels (e.g., A/B/C with a legend) so that one position equals one label.

Before you move to 3.x, measure whether temperature and logprobs still work

This is the part that needs the most care right now. The migration checklist for Gemini 3.7 Flash says plainly to strip temperature, top_p, and top_k from your generation configs, and notes that Gemini 3.6 Flash already stopped supporting them (What's new in Gemini 3.7 Flash).

The way it fails is what makes this awkward. Per the Gemini 3.6 Flash model reference, custom values for temperature, top-K, and top-P are simply ignored. Frequency penalty and presence penalty are also unsupported — but those raise an error. Same category of "not supported", two completely different signals: one breaks loudly, the other passes quietly.

Everything here rests on temperature=0. Swap the model ID to a 3.x one and your requests still return 200, and your (label, confidence) CSV still fills up exactly as before. Nothing in the output says the determinism assumption is gone. A config key that gets silently dropped is the kind of bug that eats an afternoon, so it's worth spending five minutes on it at migration time instead.

Reports on whether response_logprobs comes back at all differ from model to model. Rather than reading specs and hoping, it is faster and more reliable to fire one request at the model ID you plan to move to. The whole premise of this article is deciding by measurement rather than by feel, so the capability check should work the same way.

def probe_model(model_id: str, sample_text: str, runs: int = 8):
    """Measure whether logprobs come back and whether output is stable."""
    prompt = (
        "Classify the following email into exactly one of "
        "{inquiry, complaint, other}.\n"
        f"Email: {sample_text}\nAnswer (one word only):"
    )
 
    # 1. Do we get logprobs at all?
    try:
        resp = client.models.generate_content(
            model=model_id,
            contents=prompt,
            config=types.GenerateContentConfig(
                response_logprobs=True, logprobs=3, max_output_tokens=4
            ),
        )
        has_logprobs = resp.candidates[0].logprobs_result is not None
    except Exception as e:
        return {
            "model": model_id,
            "logprobs": f"ERROR: {type(e).__name__}",
            "deterministic": None,
        }
 
    # 2. Is temperature=0 actually landing? Count distinct outputs for one input.
    outputs = set()
    for _ in range(runs):
        r = client.models.generate_content(
            model=model_id,
            contents=prompt,
            config=types.GenerateContentConfig(temperature=0, max_output_tokens=4),
        )
        outputs.add((r.text or "").strip())
 
    return {
        "model": model_id,
        "logprobs": has_logprobs,
        "deterministic": len(outputs) == 1,
        "distinct_outputs": sorted(outputs),
    }

I keep this probe on the same branch as the commit that rewrites the model ID. A deterministic of False means temperature=0 never reached the model. Be careful with the sample you feed it, though: on a task with few label options, outputs can line up by coincidence even when the parameter is being dropped. Pick a borderline input — a row from your own logs where confidence landed between 0.6 and 0.8 makes an excellent probe. Test with something the model is sure about and you will get a pass that tells you nothing.

If you accept that temperature is off the table on 3.x, determinism has to come from the system instruction instead: spell out the output format and the boundary between labels in words. That is unglamorous work, and it happens to be the same fix as when no threshold gives you an acceptable error rate. Tightening the prompt definition pays off in both directions.

A concrete next step

Before designing anything elaborate, run response_logprobs=True over 100 representative inputs from your actual domain and dump (label, confidence) to a CSV. Plot the histogram. The point where "below this is risky" lives is usually obvious within a few minutes of looking. Once you have that number, drop it into a classify_with_gate-style function and you've meaningfully raised your pipeline's quality without changing models. Logprobs are quiet and unglamorous, but in my experience they're one of the highest-leverage flags in the Gemini API.

One last note from experience: don't try to interpret confidence in absolute terms across models. A 0.92 from gemini-2.5-flash and a 0.92 from gemini-2.5-pro are not directly comparable — different models have different temperature-zero distributions. If you upgrade the model under your gate, recalibrate the threshold against the same 100-example set — and if the upgrade crosses a model generation, run probe_model first. Confirm you can still take the measurement before you spend time re-taking it.