If you've tried building a voice AI app with Expo and hit a wall at "I understand WebSockets in theory, but how do I actually wire this up in React Native?"—you're not alone. The Web has the WebAudio API, but Expo is a different world. Figuring out expo-av, converting recorded audio into the right PCM byte format, and keeping a WebSocket session alive can easily eat up half a day of research.
This guide cuts straight to working code. The goal: get from zero to a functional real-time voice conversation with Gemini in about 30 minutes.
As an indie developer who has shipped mobile apps for years, the audio layer in Expo is exactly where I keep getting stuck — so this is the sequence that actually worked for me, with the dead ends left out.
How Gemini Live API Works
Before diving in, a quick overview of what makes Live API different from standard Gemini requests.
The standard Gemini API is a request-response pattern — you send a prompt and wait for the result. Live API uses a bidirectional WebSocket stream, where you continuously send audio and receive audio responses in return. It even supports interruptions, making conversations feel more natural.
The expected audio format is PCM 16-bit / 16 kHz / mono. If you can get Expo to record in that format and pipe it to the WebSocket, the Live API handles speech recognition, reasoning, and speech synthesis all in one step.
For a deeper look at the API specification, see the Gemini Live API Guide.
Environment Setup
Required Packages
# Create a new Expo project (skip if you already have one)
npx create-expo-app@latest gemini-voice-app --template blank-typescript
cd gemini-voice-app
# Audio recording and playback
npx expo install expo-av
# Environment variable management (to protect your API key)
npm install react-native-dotenvMicrophone Permissions
Add the following to your app.json:
{
"expo": {
"plugins": [
[
"expo-av",
{
"microphonePermission": "This app uses the microphone for voice assistant features."
}
]
]
}
}On iOS this will automatically populate Info.plist. On Android, confirm that RECORD_AUDIO is included in android.permissions in your app.json.
Implementing the WebSocket Connection
Connecting to Live API requires a bit more ceremony than a plain WebSocket. You open the connection, send a session setup message, and only then start streaming audio.
// src/hooks/useGeminiLive.ts
import { useRef, useCallback, useState } from 'react';
const GEMINI_API_KEY = process.env.EXPO_PUBLIC_GEMINI_API_KEY ?? '';
const MODEL = 'gemini-2.0-flash-live-001';
const WS_URL = `wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContent?key=${GEMINI_API_KEY}`;
type SessionStatus = 'idle' | 'connecting' | 'active' | 'error';
export function useGeminiLive() {
const ws = useRef<WebSocket | null>(null);
const [status, setStatus] = useState<SessionStatus>('idle');
const [transcript, setTranscript] = useState<string>('');
const connect = useCallback(() => {
setStatus('connecting');
ws.current = new WebSocket(WS_URL);
ws.current.onopen = () => {
// Send the session configuration as the very first message
const setupMessage = {
setup: {
model: `models/${MODEL}`,
generationConfig: {
responseModalities: ['AUDIO'],
speechConfig: {
voiceConfig: {
prebuiltVoiceConfig: { voiceName: 'Aoede' },
},
},
},
systemInstruction: {
parts: [{ text: 'You are a helpful assistant. Keep responses concise.' }],
},
},
};
ws.current?.send(JSON.stringify(setupMessage));
};
ws.current.onmessage = (event) => {
handleServerMessage(event.data);
};
ws.current.onerror = () => setStatus('error');
ws.current.onclose = () => setStatus('idle');
}, []);
const handleServerMessage = (data: string) => {
try {
const msg = JSON.parse(data);
// Session is ready
if (msg.setupComplete) {
setStatus('active');
return;
}
// Audio and text parts from the model
const parts = msg.serverContent?.modelTurn?.parts ?? [];
for (const part of parts) {
if (part.inlineData?.mimeType === 'audio/pcm;rate=24000') {
// Base64-encoded PCM audio from the model
playAudioChunk(part.inlineData.data);
}
if (part.text) {
setTranscript((prev) => prev + part.text);
}
}
} catch (e) {
console.warn('Failed to parse message:', e);
}
};
const sendAudio = useCallback((base64Audio: string) => {
if (ws.current?.readyState !== WebSocket.OPEN) return;
const message = {
realtimeInput: {
mediaChunks: [
{
mimeType: 'audio/pcm;rate=16000',
data: base64Audio,
},
],
},
};
ws.current.send(JSON.stringify(message));
}, []);
const disconnect = useCallback(() => {
ws.current?.close();
ws.current = null;
setStatus('idle');
}, []);
return { connect, disconnect, sendAudio, status, transcript };
}The playAudioChunk function is implemented in a later section. For now, this gives us the full WebSocket skeleton.
Implementing Audio Recording
Next, we need to capture microphone audio and stream it to the API.
expo-av's Audio.Recording is designed to save a file after recording finishes, but for real-time streaming we want to send chunks as we go. The approach here is to loop short recordings — record for 500ms, extract the data, send it, and immediately start the next recording.
// src/hooks/useAudioRecorder.ts
import { Audio } from 'expo-av';
import { useState, useRef, useCallback } from 'react';
import * as FileSystem from 'expo-file-system';
export function useAudioRecorder(onChunk: (base64: string) => void) {
const recording = useRef<Audio.Recording | null>(null);
const intervalRef = useRef<ReturnType<typeof setInterval> | null>(null);
const [isRecording, setIsRecording] = useState(false);
const startRecording = useCallback(async () => {
const { status } = await Audio.requestPermissionsAsync();
if (status !== 'granted') {
console.warn('Microphone permission not granted');
return;
}
await Audio.setAudioModeAsync({
allowsRecordingIOS: true,
playsInSilentModeIOS: true,
});
const rec = new Audio.Recording();
await rec.prepareToRecordAsync({
android: {
extension: '.wav',
outputFormat: Audio.AndroidOutputFormat.DEFAULT,
audioEncoder: Audio.AndroidAudioEncoder.DEFAULT,
sampleRate: 16000,
numberOfChannels: 1,
bitRate: 256000,
},
ios: {
extension: '.wav',
audioQuality: Audio.IOSAudioQuality.HIGH,
sampleRate: 16000,
numberOfChannels: 1,
bitRate: 256000,
linearPCMBitDepth: 16,
linearPCMIsBigEndian: false,
linearPCMIsFloat: false,
},
web: {},
});
await rec.startAsync();
recording.current = rec;
setIsRecording(true);
// Every 500ms: stop, read, send, restart
intervalRef.current = setInterval(async () => {
if (!recording.current) return;
await recording.current.stopAndUnloadAsync();
const uri = recording.current.getURI();
if (uri) {
const base64 = await FileSystem.readAsStringAsync(uri, {
encoding: FileSystem.EncodingType.Base64,
});
// Strip the 44-byte WAV header; send raw PCM only
const pcmBase64 = stripWavHeader(base64);
onChunk(pcmBase64);
}
// Start the next chunk recording
const next = new Audio.Recording();
await next.prepareToRecordAsync({ /* same config */ } as any);
await next.startAsync();
recording.current = next;
}, 500);
}, [onChunk]);
const stopRecording = useCallback(async () => {
if (intervalRef.current) clearInterval(intervalRef.current);
await recording.current?.stopAndUnloadAsync();
recording.current = null;
setIsRecording(false);
}, []);
return { startRecording, stopRecording, isRecording };
}
// Remove the 44-byte WAV header, return raw PCM as Base64
function stripWavHeader(base64wav: string): string {
const binary = atob(base64wav);
const pcmBinary = binary.slice(44);
return btoa(pcmBinary);
}The 500ms loop is a practical middle ground. Going shorter than 250ms tends to cause file I/O issues on Android. I've found 300–500ms works reliably across both platforms.
Implementing Audio Playback
The model returns Base64-encoded PCM at 24 kHz. To play it back with expo-av, the cleanest approach is to write a temporary WAV file and load it with Audio.Sound.
// src/utils/audioPlayer.ts
import { Audio } from 'expo-av';
import * as FileSystem from 'expo-file-system';
let soundRef: Audio.Sound | null = null;
export async function playAudioChunk(base64pcm: string) {
try {
const wav = addWavHeader(base64pcm, 24000);
const tmpUri = FileSystem.cacheDirectory + `chunk_${Date.now()}.wav`;
await FileSystem.writeAsStringAsync(tmpUri, wav, {
encoding: FileSystem.EncodingType.Base64,
});
if (soundRef) {
await soundRef.unloadAsync();
soundRef = null;
}
const { sound } = await Audio.Sound.createAsync({ uri: tmpUri });
soundRef = sound;
await sound.playAsync();
} catch (e) {
console.warn('Audio playback error:', e);
}
}
function addWavHeader(base64pcm: string, sampleRate: number): string {
const pcm = atob(base64pcm);
const dataLength = pcm.length;
const buffer = new ArrayBuffer(44 + dataLength);
const view = new DataView(buffer);
const writeString = (offset: number, str: string) => {
for (let i = 0; i < str.length; i++) {
view.setUint8(offset + i, str.charCodeAt(i));
}
};
writeString(0, 'RIFF');
view.setUint32(4, 36 + dataLength, true);
writeString(8, 'WAVE');
writeString(12, 'fmt ');
view.setUint32(16, 16, true);
view.setUint16(20, 1, true); // PCM format
view.setUint16(22, 1, true); // Mono
view.setUint32(24, sampleRate, true);
view.setUint32(28, sampleRate * 2, true); // Byte rate
view.setUint16(32, 2, true);
view.setUint16(34, 16, true); // Bit depth
writeString(36, 'data');
view.setUint32(40, dataLength, true);
for (let i = 0; i < dataLength; i++) {
view.setUint8(44 + i, pcm.charCodeAt(i));
}
const bytes = new Uint8Array(buffer);
let binary = '';
for (let i = 0; i < bytes.length; i++) {
binary += String.fromCharCode(bytes[i]);
}
return btoa(binary);
}Putting It All Together
Here's the main screen component that wires everything up:
// App.tsx
import React, { useEffect } from 'react';
import { View, Text, TouchableOpacity, StyleSheet, ScrollView } from 'react-native';
import { useGeminiLive } from './src/hooks/useGeminiLive';
import { useAudioRecorder } from './src/hooks/useAudioRecorder';
export default function App() {
const { connect, disconnect, sendAudio, status, transcript } = useGeminiLive();
const { startRecording, stopRecording, isRecording } = useAudioRecorder(sendAudio);
useEffect(() => {
connect();
return () => disconnect();
}, []);
const handlePress = () => {
if (isRecording) {
stopRecording();
} else {
startRecording();
}
};
const statusLabel: Record<string, string> = {
idle: 'Disconnected',
connecting: 'Connecting...',
active: 'Ready',
error: 'Connection error',
};
return (
<View style={styles.container}>
<Text style={styles.status}>Status: {statusLabel[status]}</Text>
<ScrollView style={styles.transcript}>
<Text style={styles.transcriptText}>
{transcript || 'Tap the button and start speaking. Transcript appears here.'}
</Text>
</ScrollView>
<TouchableOpacity
style={[styles.button, isRecording && styles.buttonActive]}
onPress={handlePress}
disabled={status !== 'active'}
>
<Text style={styles.buttonText}>
{isRecording ? '● Speaking (tap to stop)' : 'Tap to speak'}
</Text>
</TouchableOpacity>
</View>
);
}
const styles = StyleSheet.create({
container: { flex: 1, backgroundColor: '#1a1a2e', padding: 20, paddingTop: 60 },
status: { color: '#888', fontSize: 13, marginBottom: 10 },
transcript: { flex: 1, backgroundColor: '#16213e', borderRadius: 12, padding: 16, marginBottom: 20 },
transcriptText: { color: '#e0e0e0', fontSize: 15, lineHeight: 24 },
button: { backgroundColor: '#0f3460', padding: 20, borderRadius: 50, alignItems: 'center' },
buttonActive: { backgroundColor: '#e94560' },
buttonText: { color: '#fff', fontSize: 16, fontWeight: '600' },
});Run npx expo start and load it via the Expo Go app to test it on your device.
Common Pitfalls
A few things to watch out for based on my experience:
Sending audio before setup completes
Calling sendAudio before setupComplete arrives will silently fail. The status flag in the hook prevents this, but make sure your UI disables the mic button until status === 'active'.
Choppy audio output
If the model's audio responses sound fragmented, you're likely receiving multiple short chunks in quick succession. Consider buffering 2–3 chunks before playing them back sequentially rather than triggering createAsync for every tiny piece.
WAV header corruption on Android
The DataView-based WAV header builder works on Hermes (Android's JS engine), but the final Base64 encoding loop can be slow with large buffers. If you start seeing garbled audio, switch to the buffer npm package for more reliable binary handling.
API key exposure in the client
Embedding an API key in a mobile app is a security risk — it can be extracted from the binary. For production, proxy the WebSocket connection through your own backend server and use ephemeral tokens. See Gemini Live API Voice Agent Guide for the recommended token-based approach.
Start with Expo Go
The code in this guide is the minimum viable implementation — npx expo start and you're talking to Gemini in real time. Experiencing voice-to-voice conversation on a mobile device is a genuinely different feeling from text chat, and it opens up a lot of creative directions.
From here, you might explore conversation history persistence, wake-word activation, or polishing the UI for a real app. For a more production-ready React Native setup with error handling and retry logic, check out the Gemini API × React Native AI Mobile App Production Guide.