Smart mode in gemini-3.5-transcribe returns no speaker labels and no word timestamps

I was reading down the Transcription modes section of the docs when the closing note stopped me. Smart mode cannot be combined with timestamp_granularities, and it cannot be combined with diarization_mode either.

What caught my attention was where that note sits. The speaker diarization section and the word timestamps section both come before the modes section. Read the page top to bottom and you learn that speaker labels are available, that word timestamps are available, and that a nicely formatted smart mode exists — and only at the end do you learn those choices are mutually exclusive.

I have a habit of pulling configuration values out of code and into external files. As an indie developer, the thing I fear most is opening a project six months later and not knowing where to change something. With transcription_config, that habit turned into the trap itself. Here is why.

The mode field accepts both a string and an object

Everything for gemini-3.5-transcribe lives in transcription_config inside generation_config. The mode field there is unusual: it accepts either a string or an object.

from google import genai
 
client = genai.Client()
audio_file = client.files.upload(file="path/to/sample.mp3")
 
# String form (smart)
interaction = client.interactions.create(
    model="gemini-3.5-transcribe",
    input=[{"type": "audio", "uri": audio_file.uri,
            "mime_type": audio_file.mime_type}],
    generation_config={
        "transcription_config": {"mode": "smart"}
    },
)
 
# Object form (verbatim + speaker labels + word timestamps)
interaction = client.interactions.create(
    model="gemini-3.5-transcribe",
    input=[{"type": "audio", "uri": audio_file.uri,
            "mime_type": audio_file.mime_type}],
    generation_config={
        "transcription_config": {
            "mode": {
                "type": "verbatim",
                "diarization_mode": "speaker",
                "timestamp_granularities": ["word"],
            }
        }
    },
)

One key, two shapes. While you are writing it by hand, this is a non-issue. It becomes a problem the moment you externalize the config so that "only the mode" can be swapped.

You change mode: smart into an object because you now want speaker labels. Later you switch back to mode: smart for readability, and the diarization_mode you had nested inside the object disappears along with it. In either direction, nothing tells you it happened. The request does not fail.

There is a second, related trap: where those two keys live. diarization_mode and timestamp_granularities go inside the mode object, not directly under transcription_config. Meanwhile custom_vocabulary and language_codes do sit directly under it. The nesting level differs between neighbors.

What smart takes away, and what verbatim costs you

It helps to hold the whole matrix in one place before you start writing config.

Verbatim has a cost of its own. The docs state plainly that enabling word-level timestamps may degrade overall transcription accuracy. Turning it on "just in case" for a use case that never reads the offsets is trading accuracy for nothing.

The endpoints differ too. The non-streaming gemini-3.5-transcribe and the Live API's gemini-3.5-transcribe-live do not expose the same feature set.

If you design a screen that shows speaker-attributed text in real time before checking this, you will be rewriting it. When speaker labels matter, the decision to use file processing rather than streaming comes first, not last.

One more thing worth knowing before you commit to verbatim: the extra data does not arrive in the place you might expect. The transcript text comes back in interaction.output_text as usual, but speaker labels and word offsets are attached as word_info annotations on the interaction content, which you have to walk to reach.

def extract_word_annotations(interaction):
    words = []
    for step in getattr(interaction, "steps", []) or []:
        for content in getattr(step, "content", []) or []:
            for annotation in getattr(content, "annotations", []) or []:
                if getattr(annotation, "type", None) == "word_info":
                    words.append(annotation)
    return words

Each annotation carries text, speaker, start_offset, and end_offset. If you enabled diarization and only ever read output_text, you will see a perfectly normal transcript with no speaker information in it — and conclude the setting did nothing. That is a second way the same class of silent failure shows up.

Catching it before the request leaves your machine

There are more constraints here than I can reliably keep in my head. So I added a small validation step that runs right after the config is assembled. It fails before the API call, which means no upload and no billing.

"""Validate transcription_config before sending it."""
from typing import Any
 
VERBATIM_ONLY = ("timestamp_granularities", "diarization_mode")
MAX_VOCAB = 1000
RECOMMENDED_VOCAB = 100
 
 
def normalize_mode(mode: Any) -> dict:
    """mode accepts a string or an object. Coerce it to a dict."""
    if mode is None:
        return {"type": "verbatim"}
    if isinstance(mode, str):
        return {"type": mode}
    if isinstance(mode, dict):
        m = dict(mode)
        m.setdefault("type", "verbatim")
        return m
    raise TypeError(f"mode must be str or dict, got {type(mode).__name__}")
 
 
def validate(tc: dict) -> list[str]:
    errors = []
    mode = normalize_mode(tc.get("mode"))
    mtype = mode.get("type")
 
    if mtype not in ("verbatim", "smart"):
        errors.append(f"unknown mode.type: {mtype!r}")
 
    if mtype == "smart":
        for key in VERBATIM_ONLY:
            if key in mode or key in tc:
                errors.append(f"{key} cannot be combined with smart (verbatim-only setting)")
 
    for key in VERBATIM_ONLY:
        if key in tc:
            errors.append(f"{key} belongs inside mode, not directly under transcription_config")
 
    vocab = tc.get("custom_vocabulary") or []
    if len(vocab) > MAX_VOCAB:
        errors.append(f"custom_vocabulary has {len(vocab)} entries (limit {MAX_VOCAB})")
    elif len(vocab) > RECOMMENDED_VOCAB:
        errors.append(f"custom_vocabulary has {len(vocab)} entries ({RECOMMENDED_VOCAB} recommended)")
 
    return errors
 
 
def audio_limit_minutes(tc: dict) -> int:
    mode = normalize_mode(tc.get("mode"))
    heavy = any(k in mode for k in VERBATIM_ONLY)
    return 30 if heavy else 60

normalize_mode exists so the validator absorbs the two-shape reality instead of pushing it onto callers. Forcing everyone to always write the object form would throw away the one-line convenience of mode: smart, which is the reason the string form exists at all.

Running it against six configurations produced this:

OK  A defaults          limit 60 min
OK  B smart only        limit 60 min
NG  C smart + speakers  limit 30 min
      - diarization_mode cannot be combined with smart (verbatim-only setting)
OK  D verbatim + spk + word  limit 30 min
NG  E wrong nesting     limit 60 min
      - diarization_mode cannot be combined with smart (verbatim-only setting)
      - diarization_mode belongs inside mode, not directly under transcription_config
NG  F vocabulary of 240 limit 60 min
      - custom_vocabulary has 240 entries (100 recommended)

Case E is the one that actually bites in externalized-config setups. You flip mode back to the string form, and diarization_mode is left stranded one level up. The API accepts that payload without complaint. Two lines from the validator told me exactly where to look.

Case F is not an error but a departure from the recommendation. You may pass up to 1,000 terms, but the guidance is around 100. Stuffing in every proper noun you can think of likely stops helping well before the hard limit.

The vocabulary check is worth keeping even though it never blocks a request. A list that has grown past a hundred entries is usually a sign that someone appended terms over months without ever removing the ones that stopped mattering. Trimming it back to the distinctive names — the ones a general model would plausibly get wrong — is a cheaper fix than tuning anything else in the config.

Enabling features cuts your audio length in half

The limit that is easiest to miss is duration. A single request to gemini-3.5-transcribe accepts up to one hour of audio. But enable speaker diarization or word timestamps and that drops to 30 minutes.

This has more design impact than the mode conflict does. "Send a 50-minute interview in one shot with speaker labels" is simply not a shape the API supports. You need to chunk the audio, then reconcile speaker identities across chunk boundaries — and nothing guarantees that spk_1 in one chunk is the same person as spk_1 in the next.

Calling audio_limit_minutes() alongside validation pulls that decision up into design time. If you already know the duration, you can reject an impossible combination immediately. Worth remembering too: attribution for three or more speakers is documented as experimental, which matters for panel discussions and multi-guest interviews.

Decide backwards, from the artifact you need

Start from the config and you will keep hitting the conflict this article is about. Reverse the order and the choices fall out on their own.

1. Decide what the output has to be. A readable meeting summary? A searchable index? A per-speaker transcript?
2. The mode follows from that. If you need offsets or speakers, verbatim. If you do not, smart. You cannot have both.
3. Then size the audio. Verbatim with features enabled caps at 30 minutes, so this is where chunking becomes a yes or a no.
4. Add custom vocabulary last. It pays off most on audio dense with proper nouns and in-house jargon, and specifying language_codes alongside it makes the vocabulary hints land more reliably.

That ordering matters more as configuration grows. For a different angle on how config that accretes over time breaks, see Evolving Gemini API Structured Output Schemas in Production — Design Notes from an Indie App. For failures that never surface as exceptions, Gemini API Production Notes — Quiet Defenses Against 429, 500, and 503 Under Real Traffic covers adjacent ground.

If your audio work eventually moves to the real-time side, Putting an AI That Answers Phones Into Production: Building a Phone Voice Agent With Gemini Live API and Twilio Media Streams lays out a streaming-first design — the kind of architecture you need precisely because speaker labels are unavailable on the Live endpoint.

Open your config file and check two things: the mode line, and the nesting level of diarization_mode. If they are out of sync, nothing is failing right now — and that is the point.

Thank you for spending time on a small check like this one.