How to Build an Audio Transcription and Summarization App with Gemini API and Python

Setup and context — Let Gemini Read Your Audio

Audio data is everywhere — meeting recordings, interviews, podcasts, voice memos — yet extracting actionable insights from these files has traditionally required specialized speech-to-text services with complex pipelines. With Gemini API's native multimodal capabilities, that changes entirely.

Gemini can process audio files directly as input, understanding not just the words spoken but the context, speaker dynamics, and key themes. Unlike conventional transcription tools, Gemini produces natural, well-structured text that goes beyond word-for-word transcription to deliver genuinely useful summaries.

In this tutorial, you'll build a complete audio transcription and summarization application using Python and the Gemini API — from basic transcription to automated meeting report generation with structured output.

Prerequisites and Setup

What You'll Need

Python 3.10 or later
A Gemini API key from Google AI Studio (see the [Gemini API Quickstart]((/articles/gemini-api/gemini-api-quickstart) guide)
Audio files to process (MP3, WAV, M4A, OGG, etc.)

Installing the SDK

pip install google-genai

The google-genai package is Google's official Python SDK for the Gemini API, providing a clean interface for all model interactions.

Setting Your API Key

Store your API key as an environment variable for security:

export GEMINI_API_KEY="YOUR_API_KEY"

How Gemini Processes Audio

Gemini is designed from the ground up as a multimodal model, meaning it can natively process audio data without any intermediate conversion steps. Here's what happens under the hood:

Direct audio input: Audio files are sent as binary data to the API
Contextual understanding: Gemini recognizes speaker intent, topic transitions, and emphasis
Natural language output: Results are formatted as natural, readable text

Supported audio formats include MP3, WAV, AIFF, AAC, OGG, and FLAC, with support for up to approximately 9.5 hours of audio per request with Gemini 3.1 Pro.

For a broader look at audio processing capabilities, check out the [Gemini API Audio Understanding Guide]((/articles/gemini-api/gemini-audio-understanding).

Step 1 — Basic Audio Transcription

Let's start with the simplest possible implementation: sending an audio file to Gemini and getting a transcription back.

import os
from google import genai
 
# Initialize the client
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
 
# Upload the audio file
audio_file = client.files.upload(file="meeting_recording.mp3")
 
# Request transcription
response = client.models.generate_content(
    model="gemini-3.1-pro",
    contents=[
        audio_file,
        "Transcribe this audio file accurately. "
        "If there are multiple speakers, distinguish between them."
    ],
)
 
print(response.text)

Expected output:

Speaker A: Today's agenda is about the new product launch schedule.
Speaker B: Sure. Let me share the current progress. The dev team is on track...
Speaker A: Thanks. How's the marketing preparation going?
Speaker C: The landing page is complete, and ad campaign settings are...

The key here is client.files.upload(), which uploads the audio file to Gemini's servers, followed by combining it with a text prompt in generate_content(). Gemini handles speaker detection automatically when instructed.

Step 2 — Adding Summarization with Structured Output

To get both a transcription and a structured summary in one call, you can leverage Gemini's JSON output mode.

import os
import json
from google import genai
from google.genai import types
 
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
 
audio_file = client.files.upload(file="meeting_recording.mp3")
 
# Prompt for structured output
prompt = """
Analyze this audio file and output the results in the following JSON format:
 
{
  "transcript": "Full transcription with speaker labels",
  "summary": "Brief summary (under 200 words)",
  "key_points": ["Point 1", "Point 2", "Point 3"],
  "action_items": ["Action item 1", "Action item 2"],
  "duration_estimate": "Estimated audio length"
}
"""
 
response = client.models.generate_content(
    model="gemini-3.1-pro",
    contents=[audio_file, prompt],
    config=types.GenerateContentConfig(
        response_mime_type="application/json",
    ),
)
 
# Parse the JSON response
result = json.loads(response.text)
print("=== Summary ===")
print(result["summary"])
print("\n=== Key Points ===")
for point in result["key_points"]:
    print(f"• {point}")
print("\n=== Action Items ===")
for item in result["action_items"]:
    print(f"☐ {item}")

Expected output:

=== Summary ===
Discussion focused on the new product launch schedule. Development is
on track, marketing materials (landing page, ads) are ready.
Team agreed on April 15 as the target launch date with final
QA review before deployment.

=== Key Points ===
• Development team progress is on schedule
• Landing page and ad campaigns are ready
• April 15 is the target launch date

=== Action Items ===
☐ Final QA testing by April 1 (QA team)
☐ Press release preparation (Tanaka)
☐ Customer support readiness check (Suzuki)

Setting response_mime_type="application/json" forces Gemini to output valid JSON, making it easy to parse and process programmatically.

Step 3 — Timestamped Transcription for Long Audio

For longer recordings, getting timestamped segments with topic detection makes the output far more useful.

import os
import json
from google import genai
from google.genai import types
 
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
 
def transcribe_with_timestamps(file_path: str) -> dict:
    """Transcribe an audio file with timestamps and topic detection."""
    audio_file = client.files.upload(file=file_path)
 
    prompt = """
    Analyze this audio and output in the following JSON format.
    Estimate timestamps as accurately as possible.
 
    {
      "segments": [
        {
          "timestamp": "00:00 - 00:30",
          "speaker": "Speaker name or Speaker A, etc.",
          "text": "What was said"
        }
      ],
      "total_speakers": number_of_speakers,
      "language": "Detected language",
      "topics": ["Topic 1", "Topic 2"]
    }
    """
 
    response = client.models.generate_content(
        model="gemini-3.1-pro",
        contents=[audio_file, prompt],
        config=types.GenerateContentConfig(
            response_mime_type="application/json",
        ),
    )
 
    return json.loads(response.text)
 
def summarize_by_topic(transcript_data: dict) -> str:
    """Generate a topic-based summary from transcript data."""
    topics = transcript_data.get("topics", [])
    segments_text = "\n".join(
        f"[{s['timestamp']}] {s['speaker']}: {s['text']}"
        for s in transcript_data["segments"]
    )
 
    prompt = f"""
    Below is a meeting transcript.
    Detected topics: {', '.join(topics)}
 
    {segments_text}
 
    Summarize the key points for each topic, then provide
    overall conclusions and action items.
    """
 
    response = client.models.generate_content(
        model="gemini-3.1-pro",
        contents=[prompt],
    )
 
    return response.text
 
# Run the pipeline
result = transcribe_with_timestamps("long_meeting.mp3")
print(f"Speakers: {result['total_speakers']}")
print(f"Language: {result['language']}")
print(f"Topics: {', '.join(result['topics'])}")
 
summary = summarize_by_topic(result)
print("\n=== Topic-Based Summary ===")
print(summary)

This two-pass approach — first transcribing with structure, then summarizing the text — lets you optimize prompts for each stage independently.

Step 4 — Batch Processing Multiple Files

When you need to process a collection of audio files efficiently, parallel execution with proper rate limiting is key.

import os
import json
import glob
from concurrent.futures import ThreadPoolExecutor, as_completed
from google import genai
from google.genai import types
 
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
 
SUPPORTED_FORMATS = {".mp3", ".wav", ".m4a", ".ogg", ".flac", ".aac"}
 
def process_single_audio(file_path: str) -> dict:
    """Process a single audio file."""
    try:
        audio_file = client.files.upload(file=file_path)
 
        prompt = """
        Analyze this audio and output in JSON format:
        {
          "file": "filename",
          "summary": "Brief summary (under 200 words)",
          "key_points": ["Point 1", "Point 2", "Point 3"],
          "language": "Detected language",
          "sentiment": "Overall tone (positive/neutral/negative)"
        }
        """
 
        response = client.models.generate_content(
            model="gemini-3.1-pro",
            contents=[audio_file, prompt],
            config=types.GenerateContentConfig(
                response_mime_type="application/json",
            ),
        )
 
        result = json.loads(response.text)
        result["file"] = os.path.basename(file_path)
        result["status"] = "success"
        return result
 
    except Exception as e:
        return {
            "file": os.path.basename(file_path),
            "status": "error",
            "error": str(e),
        }
 
def batch_process(directory: str, max_workers: int = 3) -> list[dict]:
    """Batch process all audio files in a directory."""
    audio_files = [
        f for f in glob.glob(os.path.join(directory, "*"))
        if os.path.splitext(f)[1].lower() in SUPPORTED_FORMATS
    ]
 
    print(f"Found {len(audio_files)} audio files to process")
    results = []
 
    # Parallel processing (limit workers for API rate limits)
    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        futures = {
            executor.submit(process_single_audio, f): f
            for f in audio_files
        }
        for future in as_completed(futures):
            result = future.result()
            results.append(result)
            status = "✅" if result["status"] == "success" else "❌"
            print(f"  {status} {result['file']}")
 
    return results
 
# Run batch processing
results = batch_process("./audio_files/")
 
# Print report
success = [r for r in results if r["status"] == "success"]
print(f"\nCompleted: {len(success)}/{len(results)} successful")
for r in success:
    print(f"\n--- {r['file']} ---")
    print(f"Summary: {r['summary']}")

Using ThreadPoolExecutor speeds up processing across multiple files. Keep max_workers conservative to respect API rate limits. For more advanced async patterns, see the [Gemini API Python Async Guide]((/articles/gemini-api/gemini-api-python-async-asyncio-guide).

Step 5 — Automated Markdown Report Generation

The final piece brings everything together into an automated pipeline that generates polished Markdown reports from audio recordings.

import os
import json
from datetime import datetime
from google import genai
from google.genai import types
 
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
 
def generate_meeting_report(file_path: str, output_dir: str = "./reports") -> str:
    """Generate a complete meeting report from an audio file."""
    os.makedirs(output_dir, exist_ok=True)
    audio_file = client.files.upload(file=file_path)
 
    # Request detailed analysis
    prompt = """
    This audio is a meeting recording. Output the following in JSON format:
 
    {
      "title": "Meeting title (inferred from content)",
      "date_mentioned": "Any dates mentioned",
      "participants_count": estimated_participant_count,
      "transcript": "Full transcription",
      "summary": "Summary (under 300 words)",
      "decisions": ["Decision 1", "Decision 2"],
      "action_items": [
        {"task": "Task description", "assignee": "Person", "deadline": "Due date"}
      ],
      "next_steps": ["Next step 1", "Next step 2"]
    }
    """
 
    response = client.models.generate_content(
        model="gemini-3.1-pro",
        contents=[audio_file, prompt],
        config=types.GenerateContentConfig(
            response_mime_type="application/json",
        ),
    )
 
    data = json.loads(response.text)
 
    # Build Markdown report
    now = datetime.now().strftime("%Y-%m-%d %H:%M")
    filename = os.path.basename(file_path).rsplit(".", 1)[0]
    report = f"""# {data['title']}
 
**Generated**: {now}
**Source file**: {os.path.basename(file_path)}
**Estimated participants**: {data['participants_count']}
 
## Summary
 
{data['summary']}
 
## Decisions Made
 
"""
    for decision in data.get("decisions", []):
        report += f"- {decision}\n"
 
    report += "\n## Action Items\n\n"
    report += "| Task | Assignee | Deadline |\n|---|---|---|\n"
    for item in data.get("action_items", []):
        report += f"| {item['task']} | {item['assignee']} | {item['deadline']} |\n"
 
    report += "\n## Next Steps\n\n"
    for step in data.get("next_steps", []):
        report += f"1. {step}\n"
 
    report += f"\n## Full Transcript\n\n{data['transcript']}\n"
 
    # Save to file
    output_path = os.path.join(output_dir, f"{filename}_report.md")
    with open(output_path, "w", encoding="utf-8") as f:
        f.write(report)
 
    print(f"Report saved: {output_path}")
    return output_path
 
# Run
report_path = generate_meeting_report("weekly_standup.m4a")

With this script, a single audio file produces a complete, professionally formatted meeting report — title, summary, decisions, action items, and full transcript included.

For even more advanced multimodal processing patterns — combining images, video, and PDFs alongside audio — take a look at [Gemini API Multimodal Techniques]((/articles/gemini-api/gemini-api-multimodal-techniques).

Summary

In this tutorial, you've built a complete audio transcription and summarization application using the Gemini API and Python. Starting from basic transcription, you progressively added structured JSON output, timestamped segments with topic detection, batch processing for multiple files, and automated Markdown report generation.

Gemini's native multimodal capabilities make audio processing remarkably straightforward — no separate speech-to-text services, no complex audio pipelines. Whether you're automating meeting notes, summarizing podcasts, or analyzing interview recordings, the patterns covered here give you a solid foundation to build on.