Building Voice Agents with Gemini Live API: A Basics

The launch of Gemini Live API has transformed how AI agents interact with users. Moving beyond traditional text-based exchanges, developers can now create natural voice conversations with minimal latency. This shift significantly improves user experience and opens new possibilities for voice-driven applications.

Understanding Gemini Live API

Gemini Live API leverages Gemini Flash Live, Google's latest lightweight model optimized for real-time voice interactions.

Key Features

Real-time Processing: Processes incoming audio instantly with minimal delay
Natural Conversation: Recognizes vocal emotion and tone for more human-like responses
Low Latency: Edge-optimized processing ensures fast response times
Multi-turn Dialogue: Maintains conversation context across multiple exchanges

Gemini Flash Live is purpose-built for real-time scenarios, making it more efficient than standard Gemini models for voice applications.

Getting Started

1. Obtaining Your API Key

To use Gemini Live API, you'll need a Google API key.

Visit Google AI Studio and follow these steps:

Click "Get API Key"
Create or select a project
Click "Create API Key"
Copy the generated key

export GEMINI_API_KEY="YOUR_GEMINI_API_KEY"

2. Setting Up Your Environment

Here's a Node.js setup example:

npm init -y
npm install google-generative-ai dotenv

Basic Implementation

Minimal Voice Agent

This example demonstrates a voice agent that captures microphone input and sends it to Gemini Live API:

const { GoogleGenerativeAI } = require('@google/generative-ai');
require('dotenv').config();
 
const apiKey = process.env.GEMINI_API_KEY;
const genAI = new GoogleGenerativeAI(apiKey);
 
async function createVoiceAgent() {
  const model = genAI.getGenerativeModel({
    model: 'gemini-2.0-flash-live'
  });
 
  const session = model.startSession({
    generationConfig: {
      temperature: 0.7,
      topK: 40,
      topP: 0.95
    }
  });
 
  console.log('Voice agent initialized. Start speaking...');
 
  const audioData = Buffer.from('your_audio_bytes');
 
  try {
    const response = await session.sendMessage({
      inlineData: {
        mimeType: 'audio/webm',
        data: audioData.toString('base64')
      }
    });
 
    console.log('Agent response:');
    console.log(response.response.text());
 
  } catch (error) {
    console.error('An error occurred:', error);
  }
 
  await session.end();
}
 
createVoiceAgent();

Web Application Implementation

Here's how to capture voice input in a browser and send it to Gemini Live API:

<!DOCTYPE html>
<html>
<body>
  <button id="startBtn">Start</button>
  <button id="stopBtn">Stop</button>
  <div id="response"></div>
 
  <script>
    let mediaRecorder;
    let audioChunks = [];
 
    document.getElementById('startBtn').addEventListener('click', async () => {
      const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
      mediaRecorder = new MediaRecorder(stream);
 
      mediaRecorder.ondataavailable = (event) => {
        audioChunks.push(event.data);
      };
 
      mediaRecorder.onstop = async () => {
        const audioBlob = new Blob(audioChunks, { type: 'audio/webm' });
        await sendToGeminiAPI(audioBlob);
      };
 
      mediaRecorder.start();
    });
 
    document.getElementById('stopBtn').addEventListener('click', () => {
      mediaRecorder.stop();
    });
 
    async function sendToGeminiAPI(audioBlob) {
      const audioBase64 = await blobToBase64(audioBlob);
 
      const response = await fetch('/api/gemini-voice', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({
          audio: audioBase64,
          mimeType: 'audio/webm'
        })
      });
 
      const data = await response.json();
      document.getElementById('response').innerText = data.reply;
    }
 
    function blobToBase64(blob) {
      return new Promise((resolve) => {
        const reader = new FileReader();
        reader.onloadend = () => {
          const base64 = reader.result.split(',')[1];
          resolve(base64);
        };
        reader.readAsDataURL(blob);
      });
    }
  </script>
</body>
</html>

Backend Implementation (Node.js + Express)

const express = require('express');
const { GoogleGenerativeAI } = require('@google/generative-ai');
require('dotenv').config();
 
const app = express();
app.use(express.json({ limit: '50mb' }));
 
const genAI = new GoogleGenerativeAI(process.env.GEMINI_API_KEY);
 
app.post('/api/gemini-voice', async (req, res) => {
  try {
    const { audio, mimeType } = req.body;
 
    const model = genAI.getGenerativeModel({
      model: 'gemini-2.0-flash-live'
    });
 
    const response = await model.generateContent({
      contents: [
        {
          role: 'user',
          parts: [
            {
              inlineData: {
                mimeType: mimeType,
                data: audio
              }
            }
          ]
        }
      ]
    });
 
    const reply = response.response.text();
    res.json({ reply });
 
  } catch (error) {
    console.error('Error:', error);
    res.status(500).json({ error: error.message });
  }
});
 
app.listen(3000, () => {
  console.log('Server running on port 3000');
});

Advanced Voice Agent Features

1. Multi-turn Conversation

Maintain session state across multiple exchanges:

const session = model.startSession();
 
const response1 = await session.sendMessage(message1);
const response2 = await session.sendMessage(message2);
 
await session.end();

2. Custom System Instructions

Control agent behavior with system prompts:

const model = genAI.getGenerativeModel({
  model: 'gemini-2.0-flash-live',
  systemInstruction: {
    parts: [
      {
        text: 'You are a helpful customer service agent.'
      }
    ]
  }
});

Building on the Dolice Labs apps as an indie developer, I found that with voice the silence matters more than cleverness. I return a quick acknowledgment while generating, reconnect immediately on a drop, and watch for the user starting to speak so I can interrupt my own playback — not talking over them is half of what makes it feel natural.

Wrapping up

Gemini Live API enables developers to create natural voice conversation experiences with AI agents. Start with the examples provided and customize them for your specific use case. With Gemini Flash Live's optimized performance and understanding of natural language, your users will enjoy seamless voice interactions.