Once I started handling a handful of short promo clips as an indie developer, "making the model understand the video" became the heaviest step in the whole flow. I would cut one frame per second with ffmpeg, send each image to the model for a description, then summarize everything in one more call. It worked, but a single clip took 7-9 API calls to process, and it threw away the audio track entirely.
With Omni Flash entering public preview, passing a video in as-is and letting the model understand it natively became practical. This article walks through the minimal code I used when I moved from a frame-extraction setup to a single pass, the relative measurements, and the boundary where keeping frame sampling is still the better call.
The debt in a three-stage frame-extraction pipeline
The setup I had been running was three stages: extract, describe, summarize. Written out as code, the debt shows up clearly.
import subprocess, os
from google import genai
client = genai.Client()
def extract_frames(video_path: str, fps: float = 1.0, out_dir: str = "frames") -> list[str]:
os.makedirs(out_dir, exist_ok=True)
subprocess.run([
"ffmpeg", "-i", video_path,
"-vf", f"fps={fps}", f"{out_dir}/f_%04d.jpg",
], check=True)
return sorted(os.path.join(out_dir, f) for f in os.listdir(out_dir))
def describe_video(video_path: str) -> str:
frames = extract_frames(video_path, fps=1.0)
notes = []
for i, path in enumerate(frames): # calls grow with frame count
img = client.files.upload(file=path)
r = client.models.generate_content(
model="gemini-3.5-flash",
contents=[img, f"Frame near second {i}. Describe what's shown in one line."],
)
notes.append(f"[{i}s] {r.text.strip()}")
summary = client.models.generate_content( # one more call to summarize
model="gemini-3.5-flash",
contents=["Below are per-second frame notes. Summarize the whole video in 3 lines.\n" + "\n".join(notes)],
)
return summary.textThere are three problems. First, call count grows in proportion to video length. Second, since the audio track is never looked at, anything that depends on narration or sound effects gets dropped. Third, frames are only lined up by a time index, so the model cannot reason about motion and treats the clip as a stack of stills. In my use case, that third point was the ceiling on accuracy.
The minimal setup: hand the video straight to Omni Flash
Because Omni Flash handles video natively, you upload the clip through the Files API and pass it to a single generate_content call. Combined with structured output, the downstream parsing disappears too.
import time
from google import genai
from pydantic import BaseModel
client = genai.Client()
class VideoReport(BaseModel):
summary: str
spoken_language: str
has_music: bool
safe_for_all_ages: bool
key_moments: list[str]
def understand_video(video_path: str) -> VideoReport:
f = client.files.upload(file=video_path)
# Right after upload the file is PROCESSING. Using it before ACTIVE returns 400.
while f.state.name == "PROCESSING":
time.sleep(2)
f = client.files.get(name=f.name)
if f.state.name != "ACTIVE":
raise RuntimeError(f"upload failed: {f.state.name}")
r = client.models.generate_content(
model="gemini-omni-flash-preview", # public preview; match the exact ID in the changelog
contents=[f, "Understand this video end to end and return it in the given schema."],
config={
"response_mime_type": "application/json",
"response_schema": VideoReport,
},
)
return r.parsedIt is one call. Because the model sees visuals, audio, and the passage of time in a single context, axes like has_music and spoken_language come back at once, which the frame-extraction version could never recover. I first mirrored the old output and only took a text summary, but passing response_schema and naming the axes I actually needed made the downstream branching trivial to write.
The Before/After gap is less about lines of code and more about what you're discarding. Before pays call count to throw away audio and motion; After picks both up in a single call.