The most tedious part of turning a meeting recording into minutes isn't the transcription — Gemini handles that in seconds. It's labelling who said what. I've spent more evenings than I'd like to admit scrubbing through audio just to figure out where one speaker ended and another began.
This guide walks through how to use the Gemini API's multimodal audio understanding to extract structured "who spoke when, and what they said" data, without dropping in a dedicated diarization library. Heavy-duty diarization still belongs to specialised models like pyannote, but for a quick meeting log or a podcast excerpt, Gemini alone is more than enough.
Why Gemini API can do speaker diarization at all
Gemini 2.5 Pro and Flash accept audio directly as a multimodal input. They aren't dedicated diarization models, but they can still tell speakers apart by combining timbre cues with the flow of conversation. The key thing to keep in mind: Gemini isn't doing pure voiceprint analysis. It uses both how a voice sounds and what is being said, so any time you can pass speaker names or roles into the prompt, accuracy jumps noticeably.
Where it falls apart, predictably, is on recordings full of similar-sounding voices or heavy ambient noise. Tools like pyannote.audio or AssemblyAI's Speaker Diarization extract voiceprint embeddings and cluster them mathematically — that approach wins decisively on those harder cases. Treat Gemini as the partner that excels when context can fill in for acoustic ambiguity, and you'll pick the right tool for each job.
There's also a cost angle worth raising up front. Running a dedicated diarization model means standing up infrastructure (or paying for an API like AssemblyAI per-minute), separate from your existing LLM bill. Letting Gemini handle diarization, transcription, and summarisation in a single call collapses three line items into one — meaningful for indie projects and prototypes where every dollar of cloud spend is felt directly.
Minimal code — return labelled speaker segments as JSON
Below is the smallest working setup with the Python SDK (google-generativeai). It loads a local audio file and returns a list of utterances tagged with speaker labels and timestamps. Python 3.10 or newer is assumed.
# pip install google-generativeai>=0.8.0
import os
import json
import google.generativeai as genai
genai.configure(api_key=os.environ["GEMINI_API_KEY"])
AUDIO_PATH = "meeting.m4a" # m4a / mp3 / wav / flac / ogg are all supported
# 1) Upload the audio (File API is required for files larger than 25 MB)
audio_file = genai.upload_file(AUDIO_PATH)
# 2) Passing a participant list dramatically improves identification accuracy
participants = ["Tanaka (PM)", "Sato (Engineer)", "Lee (Designer)"]
prompt = f"""
You are a meeting diarization assistant.
Participants: {", ".join(participants)}
Task:
- List every utterance in chronological order
- Pick the speaker from the participant list above; use "unknown" when not confident
- For each utterance, include the start time in seconds and the spoken text
- Skip background noise and short fillers like "uh-huh"
Return only this JSON schema (no extra characters before or after):
{{
"segments": [
{{ "speaker": "...", "start_sec": 0.0, "text": "..." }}
]
}}
"""
model = genai.GenerativeModel(
"gemini-2.5-pro",
generation_config={
"temperature": 0.2,
"response_mime_type": "application/json",
},
)
response = model.generate_content([prompt, audio_file])
data = json.loads(response.text)
for seg in data["segments"][:5]:
print(f"[{seg['start_sec']:>6.1f}s] {seg['speaker']}: {seg['text']}")Setting response_mime_type to application/json keeps the model from prefixing its answer with explanatory text — you get clean JSON every time. Expected output looks like this:
[ 0.0s] Tanaka (PM): Thanks for joining. Let's start today's review.
[ 8.4s] Sato (Engineer): Last week's benchmark came in 20% under our cost target.
[ 21.7s] Lee (Designer): The flow updates from last review are already merged.
[ 35.2s] Tanaka (PM): Great. Let's walk through the release criteria.
[ 47.0s] unknown: (cough)
Two practical notes: keep the temperature low (the segmentation gets noticeably more stable), and route any audio above 25 MB through the File API guide so the upload succeeds.
Three prompt tweaks that meaningfully lift accuracy
Of all the things I tried in production, these three were the ones that consistently moved the needle — each one in the direction of "give the model less room to guess."
Always supply a participant list. Anonymous labels like "Speaker A / B / C" tend to drift; the model occasionally swaps identities mid-conversation. Including names and roles (PM, Engineer, Sales) gives the model a way to use domain language as a tiebreaker, and you'll see the swap rate drop immediately. If you don't have full names, even role-based hints like "two engineers and one product manager" already help.
Allow "unknown" as a label. When you force the model to assign every utterance to a known speaker, it overcommits and fabricates labels. Giving it "unknown" as an escape hatch sacrifices a few low-confidence segments but raises the precision of everything else. In my experience, around 5–10% of segments end up tagged unknown on a clean recording, and reviewing only those by ear is far cheaper than auditing every segment.
Cap segments at 30 minutes. Gemini 2.5 Pro accepts long audio, but in my experience speaker confusion ramps up past the half-hour mark. Splitting long meetings by agenda item and merging the JSON afterwards is far more reliable. Pair the schema with the JSON mode and structured output guide for stable parsing.
Cleaning up the output before it reaches your minutes
Gemini's diarization is rarely 100% clean on first pass. The two issues I see most often are: (1) a single sentence split across two segments because the speaker paused mid-thought, and (2) consecutive segments mistakenly attributed to different speakers when it's actually the same person continuing. Both are easy to fix with a small post-processing step.
def merge_consecutive_same_speaker(segments, max_gap_sec=2.0):
if not segments:
return segments
merged = [segments[0]]
for seg in segments[1:]:
prev = merged[-1]
same_speaker = seg["speaker"] == prev["speaker"]
within_gap = seg["start_sec"] - prev["start_sec"] <= max_gap_sec * 10 # heuristic
if same_speaker and within_gap:
prev["text"] = prev["text"].rstrip(".") + " " + seg["text"]
else:
merged.append(seg)
return merged
cleaned = merge_consecutive_same_speaker(data["segments"])This single helper removes the most common visual stutter in the final transcript. If you also want to detect identity swaps (the model briefly labelling someone with the wrong name), a useful trick is to run the same audio through Gemini twice with shuffled participant order in the prompt and compare the results — a stable label across both runs is far more trustworthy than a single pass.
When to choose Gemini vs. dedicated diarization tools
I get asked which to pick — pyannote or Gemini — almost weekly. My rule of thumb:
- Gemini API works well when participants are few (2–4), names and roles are known up front, context matters more than acoustics, you want one vendor for everything, and you need something running today
- pyannote / AssemblyAI win when the speaker count is unknown or large (5+), voices are similar, audio is long-form and machine-processed, or you need true voiceprint clustering accuracy
For internal meetings or small interviews, Gemini alone clears the bar. For call-centre logs or event recordings, the most reliable pattern I've found is hybrid: run dedicated diarization to cut speaker segments, then send each segment to Gemini for summarisation and entity extraction. That keeps the strengths of both — clustering precision from a specialist, contextual cleanup from Gemini — without forcing one tool to do work it isn't built for.
If you want to chain transcription, diarization, and summarisation into a single pipeline, combine this article with the audio understanding fundamentals guide and the Python audio transcription and summarisation walkthrough. They cover the surrounding pieces you'll want before pushing this into production.
Wrapping up — try it on a 30-minute recording today
Speaker diarization has a reputation for needing specialised infrastructure, but for meetings and interviews where you already know the participants, the Gemini API alone gets you to a usable place. Start with a 30-minute recording you have on hand and run the snippet above — once with the participant list, once without. The accuracy gap will tell you everything you need to know about how this kind of prompt actually shapes Gemini's output.
From there, only swap in pyannote or AssemblyAI for the parts of your workload where Gemini's contextual approach struggles (large speaker counts, similar voices, long-form audio). Most teams I've seen go this route end up with a leaner, cheaper pipeline than they started with.
One last suggestion: log every output you ship to production, alongside the participant list and audio length you used. After a few weeks you'll have a small dataset that tells you exactly where Gemini's diarization holds up and where it doesn't, in the specific contexts you actually care about. That kind of grounded calibration is far more useful than any benchmark, because the failure cases that matter to you are rarely the ones papers measure. Keep it close to the work, and the right tool for each segment of audio will become obvious without you having to guess.