Stop Making Listeners Wait for the Whole File — Wiring Gemini TTS Streaming into Your Delivery Path

The WAV Header Problem — Choosing a Format When the Length Isn't Known Yet

A WAV header contains a data-length field. In streaming generation the total duration is unknown until the very end, so you cannot write a correct header up front. I evaluated three options.

1. Write a fake maximum length into the header and feed it to an <audio> tag. Works in some browsers, but duration display breaks and I hit playback stalls in Safari-family browsers. Rejected.
2. Transcode to MP3 on the fly server-side. Requires a resident ffmpeg, a dependency I did not want to bring into my Cloudflare Workers-centric setup. Rejected.
3. Deliver raw PCM and play it with the Web Audio API on the client. The header problem simply disappears. Adopted.

Here is the client. There is one pitfall I only discovered by implementing it: HTTP chunk boundaries do not respect Int16 boundaries. A chunk can end on an odd byte; read it straight into an Int16Array and you get an exception — or worse, one-sample-shifted noise. A one-byte carry-over buffer absorbs it.

// player.js — play raw PCM as it arrives (with Int16 boundary carry-over)
async function playStream(text) {
  const res = await fetch(`/tts?text=${encodeURIComponent(text)}`);
  const reader = res.body.getReader();
  const ctx = new AudioContext({ sampleRate: 24000 });
  let playhead = ctx.currentTime + 0.3; // 300ms initial buffer
  let carry = new Uint8Array(0);        // odd-byte carry-over
 
  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
 
    const merged = new Uint8Array(carry.length + value.length);
    merged.set(carry); merged.set(value, carry.length);
    const usable = merged.length - (merged.length % 2);
    carry = merged.slice(usable);
 
    const pcm = new Int16Array(merged.buffer, 0, usable / 2);
    const buf = ctx.createBuffer(1, pcm.length, 24000);
    const ch = buf.getChannelData(0);
    for (let i = 0; i < pcm.length; i++) ch[i] = pcm[i] / 32768;
 
    const src = ctx.createBufferSource();
    src.buffer = buf;
    src.connect(ctx.destination);
    src.start(playhead);
    playhead += buf.duration;
  }
}

The 300ms initial buffer exists to absorb jitter in chunk arrival. At 100ms I got an audible click every few dozen seconds in my environment; at 300ms it disappeared. This number depends on your network and region, so measure on real devices before tightening it.

Failure Semantics — Resume at Sentence Boundaries, Not Byte Offsets

This is the design decision unique to streaming. With batch, failure means "regenerate everything" and nobody notices. With streaming, the listener has already heard part of the audio; restarting from the top is the worst possible experience.

My first instinct was byte-offset resume. I abandoned it quickly: generation is not deterministic, so a second render of the same text does not match the first one byte-for-byte. Offset-based resume cannot work even in principle.

So I raised the resume unit to the sentence. Split the draft into sentences; the client only remembers how many sentences it has finished playing. On reconnect, send the remainder of the draft.

# resume.py — sentence-boundary resume
import re
 
def split_sentences(text: str) -> list[str]:
    return [s for s in re.split(r"(?<=[.!?])\s+", text) if s.strip()]
 
@app.get("/tts/resume")
def tts_resume(text: str, done_sentences: int = 0):
    sentences = split_sentences(text)
    remaining = " ".join(sentences[done_sentences:])
    if not remaining:
        return Response(status_code=204)
    return StreamingResponse(
        stream_tts(remaining),
        media_type="audio/L16;rate=24000;channels=1",
    )

The client advances done_sentences by accumulating estimated per-sentence duration (derived from character counts — see the coefficient below). Resume does replay a little (1.4 sentences on average), which is well within acceptable compared to starting over.

Measured Numbers, and Insurance for Running a Preview Model in Production

Measured across ten 3,800-character article drafts:

• Time to first sound: batch averaged 41.2s → streaming averaged 1.8s (roughly 23x)
• Chunk arrival interval: median ~180ms, p95 ~420ms (the basis for the 300ms initial buffer)
• Duration estimation: my drafts land within ±5% of measured audio length using a flat characters-per-second coefficient — useful for missing-audio detection as well
• Disconnect rate: about 2% on connections longer than five minutes; with sentence-boundary resume, only 1.4 sentences get regenerated on average
• Cost: the amount of audio generated is identical, so I observed no cost difference between batch and streaming

One last thing I would not gloss over: this is a preview model. In June I had an automation stop when the image preview models were shut down on 6/25, and since then I write the escape route before putting any preview model on a production path.

# fallback.py — degrade to batch when the preview goes away
def tts_with_fallback(text: str):
    try:
        yield from stream_tts(text)
    except Exception as e:
        if "NOT_FOUND" in str(e) or "deprecated" in str(e).lower():
            # streaming unavailable — fall back to batch (slower, but the path survives)
            audio = batch_tts(text)  # your existing batch implementation
            yield audio
        else:
            raise

Keep the model name in config, and health-check streaming availability once at startup. Those two habits turn a sudden preview shutdown into a degradation instead of an outage.

Start by leaving your batch TTS untouched and swapping only the listen-button path over to this setup. First sound in under two seconds changes the experience more than the number suggests. I hope this saves you a rebuild or two.