Generating Multilingual Video Subtitles (SRT/VTT) with the Gemini API

Adding subtitles to a short video sounds simple — until you try to automate it. The traditional pipeline runs through three or four moving parts: Whisper for transcription, a translation API for the second language, and finally a script that wraps everything in SRT timestamps. It works, but every joint is a place where things can drift out of sync.

The Gemini API's multimodal video input lets you collapse that whole pipeline into a single request. The catch is that "it runs" and "it's good enough to ship" are not the same thing. After a few weekends of trial and error on my own short clips, this is the pattern I keep coming back to for five-minute videos.

Two ways to feed video to the Gemini API

There are two paths for sending video to Gemini, and the right choice depends mostly on file size.

Inline: Base64-encode the file and embed it directly in parts. Fine for clips up to roughly 20 MB
File API: Upload with client.files.upload() and reference the resulting file. Supports up to 2 GB and stays cached for 48 hours

A five-minute phone recording is easily 50–80 MB, so in practice you'll want the File API for almost any real workload. The one quirk: uploads start in PROCESSING state and only become usable once they reach ACTIVE. You need to poll for that.

from google import genai
from google.genai import types
import time
 
client = genai.Client(api_key="YOUR_API_KEY")
 
# Upload the video
video_file = client.files.upload(file="meeting.mp4")
print(f"Uploaded: {video_file.name}, state: {video_file.state}")
 
# Poll until the file is ACTIVE
while video_file.state == "PROCESSING":
    time.sleep(2)
    video_file = client.files.get(name=video_file.name)
print(f"Ready: {video_file.state}")  # expected: ACTIVE

If you send a request while the file is still PROCESSING, the API returns 400 INVALID_ARGUMENT. Always wait for ACTIVE first.

Three tricks for tighter timestamps

The first wall most people hit is timestamp drift. Gemini gives you reasonable timings out of the box, but they tend to wander 1–3 seconds off without a bit of nudging.

Three things made the biggest difference for me:

Specify the sampling fps. The default is 1 fps. Bumping video_metadata.fps up to 2–4 dramatically reduces missed lines on quick speech
Ask for MM:SS.mmm format explicitly. Saying "millisecond precision" in the prompt nudges the model to track time more carefully
Cap caption length per cue. Telling the model "max 35 characters per cue, two lines" prevents one giant caption from staying on screen for 12 seconds

config = types.GenerateContentConfig(
    response_mime_type="application/json",
    response_schema={
        "type": "ARRAY",
        "items": {
            "type": "OBJECT",
            "properties": {
                "start": {"type": "STRING", "description": "MM:SS.mmm format"},
                "end": {"type": "STRING", "description": "MM:SS.mmm format"},
                "ja": {"type": "STRING"},
                "en": {"type": "STRING"},
            },
            "required": ["start", "end", "ja", "en"]
        }
    }
)
 
prompt = """Transcribe the spoken audio in this video into subtitle cues.
Requirements:
- Timestamps in MM:SS.mmm format (e.g. 01:23.450)
- Max 35 characters per cue, max two lines (use \\n for line break)
- 'ja' is the original Japanese speech verbatim; 'en' is a natural English translation
- Skip any silent or BGM-only segments
"""
 
response = client.models.generate_content(
    model="gemini-2.5-pro",
    contents=[video_file, prompt],
    config=config,
)
 
import json
captions = json.loads(response.text)
print(f"Generated {len(captions)} cues")

Using response_schema guarantees JSON output, which makes the next step — converting to SRT — pleasantly mechanical.

Converting to SRT and VTT

SRT and VTT differ in two small but annoying ways: SRT separates milliseconds with a comma, VTT uses a period, and VTT requires a WEBVTT header at the top.

def to_srt(captions: list[dict]) -> str:
    """captions: [{start, end, ja, en}, ...]"""
    lines = []
    for i, c in enumerate(captions, 1):
        start = c["start"].replace(".", ",")
        end = c["end"].replace(".", ",")
        # Promote MM:SS,mmm to HH:MM:SS,mmm
        start = f"00:{start}" if start.count(":") == 1 else start
        end = f"00:{end}" if end.count(":") == 1 else end
        lines.append(f"{i}\n{start} --> {end}\n{c['ja']}\n")
    return "\n".join(lines)
 
def to_vtt(captions: list[dict], lang: str = "ja") -> str:
    body = []
    for c in captions:
        start = f"00:{c['start']}" if c['start'].count(":") == 1 else c['start']
        end = f"00:{c['end']}" if c['end'].count(":") == 1 else c['end']
        body.append(f"{start} --> {end}\n{c[lang]}")
    return "WEBVTT\n\n" + "\n\n".join(body)
 
with open("output_ja.srt", "w") as f:
    f.write(to_srt(captions))
with open("output_en.vtt", "w") as f:
    f.write(to_vtt(captions, lang="en"))

YouTube accepts both. For HTML5 <track> tags on your own site, VTT is the only valid choice.

Generate every language in one shot

Accessibility captions in the original language plus a translated track for international viewers — that's the most common combo for short-form content. The intuitive approach is "transcribe first, then translate," but it leaves you with two passes that don't share timestamps. Cues end up subtly out of sync with the audio.

The cleaner pattern is to put both languages inside the same response_schema and let Gemini emit them together, sharing a single timing track. The earlier example does this for ja and en. Add zh-Hans, ko, or es to the schema and you get those tracks "for free" — but the catch is that output tokens scale linearly with language count. Two or three languages per request stays affordable; beyond that I'd split into separate calls.

Picking the right model for the job

In testing, Gemini 2.5 Flash transcribes a five-minute clip in roughly half the wall-clock time of 2.5 Pro and at a fraction of the cost. For casual personal content where you can spot-check the output, Flash is the obvious starting point. Where Pro earns its keep is on speakers with strong accents, mixed-language audio, or footage with a lot of background noise — Pro is noticeably better at separating the primary voice from ambient sound.

A pragmatic workflow is to default to Flash for first drafts, review a representative sample, and re-run on Pro only when the Flash output has too many errors to fix manually. The same response_schema and prompt work on both models, so swapping is a one-line change:

# Cheap first pass
response = client.models.generate_content(model="gemini-2.5-flash", contents=[video_file, prompt], config=config)
# Re-run on Pro for noisy footage
response = client.models.generate_content(model="gemini-2.5-pro", contents=[video_file, prompt], config=config)

Cost-wise, on the clips I tested, Flash was running at roughly 10–15% of the Pro cost for the same length input. That gap matters once you start processing dozens of videos a week.

A lightweight QA pass before publishing

Even with a tight prompt, machine-generated subtitles benefit from a quick sanity check before they go in front of viewers. Three checks catch the majority of issues:

Reading speed. Cues that flash by faster than ~17 characters per second are unreadable. Compute len(text) / (end - start) for each cue and flag anything above that threshold for manual review
Overlap detection. Adjacent cues whose end time is greater than the next start will render on top of each other. A simple linear scan catches these in milliseconds
Coverage gaps. If the longest gap between cues is more than 10 seconds, you've probably either lost a section to silence detection (false positive) or the model dropped a chunk of dialogue

def qa_captions(captions: list[dict]) -> list[str]:
    issues = []
    def to_seconds(ts: str) -> float:
        # MM:SS.mmm -> seconds
        m, rest = ts.split(":", 1)
        return int(m) * 60 + float(rest)
    prev_end = 0.0
    for i, c in enumerate(captions):
        s, e = to_seconds(c["start"]), to_seconds(c["end"])
        cps = len(c["ja"]) / max(e - s, 0.1)
        if cps > 17:
            issues.append(f"cue {i}: too fast ({cps:.1f} cps)")
        if s < prev_end:
            issues.append(f"cue {i}: overlaps previous cue")
        if s - prev_end > 10:
            issues.append(f"cue {i}: {s - prev_end:.1f}s gap before this cue")
        prev_end = e
    return issues
 
for problem in qa_captions(captions):
    print(problem)

I run this automatically as part of the generation script and only escalate to manual review when the issue list is non-empty. For most clean recordings the list comes back empty and the SRT is ready to upload as-is.

Pitfalls I hit in production

Videos longer than an hour need to be chunked. Even Gemini 2.5 Pro takes a long time on a one-hour file and is more likely to time out. Slice into 15–20 minute pieces, process them in parallel, and add a per-chunk time offset when stitching the SRT back together
The model invents speech in silence. Sections with only BGM occasionally come back with a fabricated "uh" or "hmm." Stating "do not output cues for silent or music-only sections" in the prompt fixes this in most cases
Proper nouns are unreliable. Product names, people's names, and your own brand name will be mistranscribed unless you supply them as a glossary at the end of the prompt: "Always spell these proper nouns exactly: Dolice, gemilab.net, ..."

Wrapping up — start with a 30-second clip

Trying to debug a one-hour video that produced bad subtitles is miserable, because you can't tell which step failed. Get the minimal pipeline — File API upload, response_schema, SRT export — working on a 30-second clip first, confirm the file plays correctly with subtitles in your video player, and only then scale up.

Once that first end-to-end run succeeds, almost all remaining quality work happens in the prompt itself. The Google AI Studio free tier is more than enough to iterate, so try it on your own footage when you have an hour to spare.