Reading Video with Gemini API — A Practical Guide to Video Understanding

Introduction

Among all the capabilities in the Gemini API, video understanding is the one that genuinely feels like the future. You hand it a video file, and the model comprehends what's happening — it answers questions, summarizes scenes, and extracts structured information from both the visual and audio tracks.

Recently, I built a small tool that automatically extracts action items from meeting recordings using this feature. The accuracy surprised me — it was solidly practical from the first test. Here's what I learned building it.

What Sets Gemini's Video Understanding Apart

Other AI models support image understanding, but Gemini stands out with its native long-form video support:

• Processes videos up to 2 hours in a single call
• Understands specific timestamps — you can ask about a particular moment
• Integrates audio and visual tracks as a unified multimodal input
• Accepts YouTube URLs directly for public videos

That last point — understanding audio and video together — is what makes it particularly powerful for meeting recordings and presentation analysis. The model doesn't just describe what it sees; it comprehends context, conversation flow, and intent.

Uploading Video with the File API

For larger video files, the recommended approach is to upload through the File API first, then reference the uploaded file in your prompts.

import google.generativeai as genai
import time
 
genai.configure(api_key="YOUR_API_KEY")
 
def upload_video(file_path: str):
    print(f"Uploading: {file_path}")
    video_file = genai.upload_file(path=file_path)
 
    # Wait for processing to complete
    while video_file.state.name == "PROCESSING":
        print("Processing...")
        time.sleep(5)
        video_file = genai.get_file(video_file.name)
 
    if video_file.state.name == "FAILED":
        raise ValueError(f"Upload failed: {video_file.state.name}")
 
    print(f"Upload complete: {video_file.uri}")
    return video_file
 
video_file = upload_video("meeting_recording.mp4")

Uploaded files are retained for 48 hours. If you're running recurring scripts, store the file URI and reuse it to avoid re-uploading the same content.

Extracting Action Items from Meeting Recordings

Here's the core use case I built — passing a meeting recording and extracting structured action items:

model = genai.GenerativeModel(model_name="gemini-2.0-flash")
 
prompt = """
Analyze this meeting recording and extract the following in structured format:
 
## Action Items
- List each item with: owner, task description, deadline
- Include timestamps where each was mentioned (e.g., [12:34])
 
## Decisions Made
- Bullet list of confirmed decisions from the meeting
 
## Open Questions
- Issues left unresolved for follow-up
 
Please format the output in Markdown.
"""
 
response = model.generate_content([video_file, prompt])
print(response.text)

This straightforward approach extracted action items from a one-hour meeting in under five minutes. The quality was high enough that we're now piloting it for our weekly team standups.

Querying Specific Timestamps

One of the more powerful features is the ability to ask about specific time ranges:

# Ask about a specific window
response = model.generate_content([
    video_file,
    "What was the most contentious topic discussed between the 30 and 45 minute marks?"
])
 
# Extract slide content throughout the video
response = model.generate_content([
    video_file,
    "List all the moments when slides changed, with the timestamp and the key message of each slide."
])

The model understands the video's timeline, so natural-language time references — "around the halfway point" or "near the end" — work surprisingly well.

Passing YouTube URLs Directly

For publicly available YouTube videos, you can pass the URL without any file upload:

response = model.generate_content([
    "https://www.youtube.com/watch?v=VIDEO_ID",
    "Summarize the main arguments in this video and identify the three most important points the presenter emphasized."
])

Keep in mind that YouTube URL support may have regional or content-based restrictions. For production workloads, uploading via the File API is the more reliable path.

What I Learned from Testing

Accuracy in Practice

Transcription accuracy for conversational speech was strong, even with technical jargon mixed in — the model uses context to handle ambiguous terms well. The one area where it struggled was distinguishing speakers when multiple people talked over each other, which is a known challenge for any audio processing system.

Choosing Between Flash and Pro

• Gemini 2.0 Flash: Faster, cheaper, and well-suited for throughput-heavy tasks like meeting summaries where you prioritize speed over deep reasoning.
• Gemini 2.5 Pro: Noticeably better at tasks requiring nuanced judgment — detecting inconsistencies, surfacing implicit assumptions, or cross-referencing claims within a long discussion.

For a one-hour video, Flash typically returns results in 3–5 minutes.

File Size Considerations

The File API upload limit is 2GB. For videos larger than that — or to speed up processing — compressing with ffmpeg before uploading is worth doing. For meeting recordings, audio clarity matters far more than video resolution:

# Downscale to 480p while preserving audio quality
ffmpeg -i input.mp4 -vf scale=-1:480 -c:a aac -b:a 128k output_compressed.mp4

Ideas Worth Exploring

The video understanding feature opens up a wide range of practical applications:

• Documentation from tutorial videos: Convert product walkthrough recordings into written step-by-step guides
• Sports analysis: Feed game footage and get tactical breakdowns or player movement analysis
• Educational content summarization: Auto-extract key concepts from recorded lectures
• Demo QA: Compare a recorded product demo against official specs to catch inconsistencies

Closing Thoughts

Gemini's video understanding feature has a low implementation barrier and high practical value. Upload a file, write a prompt describing what you want to extract, and you're running. The hardest part is usually crafting the prompt to get consistently structured output.

For any workflow where recorded information needs to be organized, retrieved, or acted on, this is a feature worth putting to work. We'll be sharing more concrete implementation examples on Gemini Lab in the coming weeks.