◈ API / SDK/2026-07-07Advanced

Extract Social Media Promo Metadata From Short Videos in One Omni Flash Pass

Hand a short clip to the public preview of Gemini Omni Flash once and get captions, chapters, and highlight timestamps back as structured JSON. Covers how this differs from a frame-extraction multi-call setup, where fps and media_resolution actually matter, and a per-clip cost estimate — from the angle of keeping an indie promo workflow moving.

gemini⁹⁶ omni-flash multimodal⁴³ video⁵ structured-output²⁰

✦ Premium Article

Every time I finish a new short video, the thing that stalls me is not the video itself — it is writing the announcement copy for it. Watching a 30-second clip over and over, noting the seconds where the good moments land, drafting a caption and hashtags. As an indie developer at Dolice, this stretch has always been the bottleneck.

With Omni Flash in public preview, that hand transcription collapses into a single API call. Instead of slicing the video into frames and sending them one by one, you hand the model the video natively and get the promo elements back as structured JSON. Here is the minimal setup, with working code and a cost estimate you can reuse.

Choosing not to slice into frames

The classic approach to video understanding was to pull a frame every few seconds with ffmpeg and send each still image separately. It is stable, but the motion between the sampled instants is lost, and every extra call turns directly into cost and latency.

Omni Flash takes one whole video natively and interprets it with the timeline intact. Where exactly to let go of frame extraction is something I worked through, measurements included, in folding video understanding into a single Omni Flash pass. This article picks up after that: turning the single-pass understanding into structured data you can actually publish with.

Promo metadata is, in fact, one of the better fits for Omni Flash within video work — because the shape of the output is decided up front. Caption, hashtags, chapters, highlight timestamps. When you can fix the output shape in advance, the most stable move is to stop the model from writing freely and pour its answer into a schema.

Decide the output shape first

Declare the JSON you want with Pydantic. Leave this vague and the model returns a different structure every run, and your downstream parsing breaks.

from pydantic import BaseModel
 
class Chapter(BaseModel):
    start: str   # mm:ss like "0:07"
    title: str   # a short heading for the chapter
 
class Highlight(BaseModel):
    timestamp: str  # the good moment, "0:12"
    reason: str     # why this is a highlight
 
class PromoMeta(BaseModel):
    hook_caption: str        # the short hook for the first line
    long_caption: str        # a slightly longer body caption
    hashtags: list[str]      # expect 5 to 8
    chapters: list[Chapter]  # the flow of the video
    highlights: list[Highlight]  # candidates for thumbnails or cutdowns

The key is that timestamp is a mm:ss string, not a number. Ask the model to turn video time into a number and the unit wobbles between seconds and frames. Take it in the form you will display, and convert to seconds later if you need to.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦If you have been hand-writing promo copy from every clip, you can now get a first draft back from a single API call

✦You will get a working design for pulling captions, chapters, and highlight timestamps as a typed object, with copy-paste code

✦You can decide how far to cut fps and media_resolution before accuracy breaks, and map a per-clip cost estimate onto your own workflow

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

Hand the video over once and take back structured metadata

Upload the video to the Files API, wait until it is ACTIVE, then make a single call with response_schema.

import time
from google import genai
from google.genai import types
 
client = genai.Client(api_key="YOUR_API_KEY")
 
video_file = client.files.upload(file="promo_clip.mp4")
 
# Right after upload the state is PROCESSING. Wait for ACTIVE.
while video_file.state.name == "PROCESSING":
    time.sleep(2)
    video_file = client.files.get(name=video_file.name)
 
prompt = (
    "This clip is a promo for a mobile app. "
    "Write the announcement copy, hashtags, chapters, and highlights for a social post. "
    "Keep hook_caption under 40 characters and long_caption around 120 characters."
)
 
resp = client.models.generate_content(
    model="gemini-omni-flash-latest",  # confirm the preview model id in the changelog
    contents=[video_file, prompt],
    config=types.GenerateContentConfig(
        response_mime_type="application/json",
        response_schema=PromoMeta,
        media_resolution=types.MediaResolution.MEDIA_RESOLUTION_LOW,
    ),
)
 
meta = resp.parsed
print(meta.hook_caption)
for h in meta.highlights:
    print(h.timestamp, h.reason)

resp.parsed is an object already validated against response_schema. When it comes back empty or None, it is usually because response_mime_type was not set to application/json, or the schema has so many required fields that the model cannot fill them all. For isolating schema-validation trouble, I wrote up the steps in fixing Gemini API structured output errors.

Model IDs shift during a public preview. When I move something to production, I keep the model name in an environment variable rather than hard-coding it. The live preview target is worth checking each time against the Gemini API changelog.

When highlight seconds drift, suspect fps

Of all the structured fields, the one whose accuracy matters most is the highlights timestamp. Let it drift by a few seconds and it stops being usable as a cutdown or thumbnail instruction.

By default the video you hand to Omni Flash is sampled at roughly one frame per second. On fast-paced clips, that granularity can miss a highlight by one or two seconds. For fast motion, raise the fps on the Part.

part = types.Part(
    file_data=types.FileData(
        file_uri=video_file.uri,
        mime_type=video_file.mime_type,
    ),
    video_metadata=types.VideoMetadata(fps=2),  # default 1 -> 2 to catch finer motion
)
 
resp = client.models.generate_content(
    model="gemini-omni-flash-latest",
    contents=[part, prompt],
    config=types.GenerateContentConfig(
        response_mime_type="application/json",
        response_schema=PromoMeta,
    ),
)

The higher the fps, the finer the timestamps — but input tokens grow almost proportionally. In my environment, raising fps from 1 to 2 on a 30-second app promo clip pulled the highlight drift from around 2 seconds down to under 1 second, while input tokens roughly doubled. The trade-off depends on the tempo of the clip. A slow, calming clip is fine at fps 1; a fast game screen wants 2 to 3. To push the fps-to-token relationship one level further, see timestamp accuracy and fps control in video understanding.

Estimating the per-clip cost

The call to automate promo work comes down to what it costs per clip. Video input tokens scale roughly as seconds x fps x tokens-per-frame.

Setting	Input tokens for a 30s clip (rough)	Use case
media_resolution LOW / fps 1	lowest	calming, slideshow-style promos
media_resolution LOW / fps 2	about 2x	app screens with motion
media_resolution MEDIUM / fps 2	higher still	clips where fine detail must be read

In my own runs, LOW was enough for media_resolution when the goal is promo metadata. Captions and hashtags do not need the fine text inside a frame. The only time detail matters is when you want the model to read UI text in the video. Start at LOW / fps 1, and raise fps only on the clips where highlights drift — that is the honest order to cut cost.

The cases that break, and how to patch them

Single-pass extraction is comfortable, but it breaks in characteristic ways. Here are the ones I hit while running my own promo pipeline.

On long videos, highlights can skew toward the first half. The model tends to decide early that it has "found the highlight." For anything over three minutes, split into two calls for the first and second halves and merge afterward for a more even spread.

On clips with long silent stretches, long_caption tends toward an abstract line about the mood alone. Adding a condition to the prompt — "include at least one concrete element visible on screen" — cut down how much I had to rewrite.

Vertical video (9:16) works too, but captions pinned to the screen edge can be missed. If information from those captions is essential to the announcement, supply that wording yourself and let the model focus on structure and timestamps.

Always run the output past a human before posting

One last thing — the line I hold to. This method helps with the work of putting a piece out into the world: the announcement copy, the hashtags. The piece itself — the app, the visual expression in the video — I still make by hand, as before.

I let AI draft the announcement, then read it back and adjust it myself before posting. That order is the one thing I do not break. When a generated long_caption drifts from my own voice, I rewrite it there. Handing announcements to AI is how I protect the time for the creative work itself — a distinction that settled into place across years of running indie development alongside an art practice.

The next step

Take one short clip you have on hand and run it through the PromoMeta schema at media_resolution LOW / fps 1. Look at how well the returned highlights timestamps line up with the actual good moments, and you will see at a glance what fps that clip needs. From there, decide whether to cut or add — and you will find the minimal setup that fits your own promo workflow.

If it lightens even a little of the time you have been losing to writing promos from video, I will be glad. Thank you for reading.

Thank You for Reading

Gemini Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.