Mastering Gemini API Streaming Responses

You built a chat UI with Gemini API's streaming response. Worked perfectly in the demo. Then users started reporting "the answer stops mid-sentence," "garbled characters," and "the same paragraph appears twice."

Getting streaming to work is easy. Making it not break is exponentially harder. This article shares lessons from running a streaming chat interface in production for a year.

Understanding Chunk Internals

Gemini API's generateContentStream returns responses split into chunks. But chunk boundaries follow no predictable pattern — they can split mid-sentence, mid-word, and even mid-character in multi-byte encodings like UTF-8.

import { GoogleGenerativeAI } from '@google/generative-ai';
 
const genAI = new GoogleGenerativeAI('YOUR_API_KEY');
const model = genAI.getGenerativeModel({ model: 'gemini-2.5-pro' });
 
const stream = await model.generateContentStream('Explain the four seasons in Japan');
 
// Inspect each chunk's structure
for await (const chunk of stream.stream) {
  console.log({
    text: chunk.text(),
    candidates: chunk.candidates?.length,
    finishReason: chunk.candidates?.[0]?.finishReason,
    // safetyRatings are included in every chunk
    safety: chunk.candidates?.[0]?.safetyRatings,
  });
}

Three facts you need to know:

text() returns the delta: Each chunk's text() contains only that chunk's text portion, not a cumulative result
finishReason exists only in the final chunk: It is undefined for all intermediate chunks
Empty chunks are normal: A chunk where text() returns an empty string is valid — it may only contain safety rating updates

Ignoring fact three is a common source of bugs where error handling triggers on empty chunks and aborts the stream prematurely.

Buffering to Prevent Garbled Text

The trickiest streaming issue with CJK text (Chinese, Japanese, Korean) is multi-byte characters being split across chunks. This becomes especially problematic when streaming over Server-Sent Events (SSE):

class StreamBuffer {
  private buffer = '';
 
  // Accept a chunk, return safely displayable text
  process(chunk: string): string {
    this.buffer += chunk;
 
    // Check for incomplete UTF-8 sequences at the end
    const lastCharCode = this.buffer.charCodeAt(this.buffer.length - 1);
    if (isHighSurrogate(lastCharCode)) {
      // High surrogate without its pair — wait for the next chunk
      return this.buffer.slice(0, -1);
    }
 
    const output = this.buffer;
    this.buffer = '';
    return output;
  }
 
  flush(): string {
    const remaining = this.buffer;
    this.buffer = '';
    return remaining;
  }
}
 
const isHighSurrogate = (code: number) => code >= 0xD800 && code <= 0xDBFF;

The Gemini Node.js SDK handles this to some extent, but if you are working with WebSocket or SSE directly, you need to implement buffering yourself.

Automatic Recovery from Disconnections

Handling mid-stream network drops or Gemini API timeouts is non-negotiable for production. Rather than a simple retry, build a "resume from where we left off" mechanism:

const streamWithRecovery = async (
  prompt: string,
  onChunk: (text: string) => void,
  maxRetries = 3
) => {
  let accumulated = '';
  let retries = 0;
 
  while (retries <= maxRetries) {
    try {
      const currentPrompt = retries === 0
        ? prompt
        : `${prompt}\n\n[Continue from where you left off. Previous response ending: "${accumulated.slice(-500)}"]`;
 
      const stream = await model.generateContentStream(currentPrompt);
 
      for await (const chunk of stream.stream) {
        const text = chunk.text();
        if (text) {
          accumulated += text;
          onChunk(text);
        }
      }
 
      // Completed successfully
      return accumulated;
    } catch (error) {
      retries++;
 
      if (retries > maxRetries) throw error;
 
      // Exponential backoff
      const delay = Math.min(1000 * Math.pow(2, retries), 10000);
      await new Promise(resolve => setTimeout(resolve, delay));
 
      // Notify user of recovery attempt
      onChunk('\n\n[Reconnecting...]\n\n');
    }
  }
 
  return accumulated;
};

The prompt design for "continuation" is the key. Including the last 500 characters gives Gemini enough context to resume without losing coherence. That said, this approach does not guarantee a seamless continuation — some duplication or stylistic discontinuity is possible. Surface this possibility in the UI so users are not surprised.

Frontend Rendering Strategy

When rendering streaming text in real time, the most important decision is when to parse Markdown:

import { useRef, useState, useCallback } from 'react';
import { marked } from 'marked';
 
const useStreamingRenderer = () => {
  const [displayHtml, setDisplayHtml] = useState('');
  const rawTextRef = useRef('');
  const renderTimerRef = useRef<NodeJS.Timeout | null>(null);
 
  const appendChunk = useCallback((chunk: string) => {
    rawTextRef.current += chunk;
 
    // Debounce: batch consecutive chunks within 50ms for a single render
    if (renderTimerRef.current) {
      clearTimeout(renderTimerRef.current);
    }
 
    renderTimerRef.current = setTimeout(() => {
      const html = marked.parse(rawTextRef.current, {
        breaks: true,
        gfm: true,
      });
      setDisplayHtml(html as string);
    }, 50);
  }, []);
 
  const finalize = useCallback(() => {
    // Final render (no debounce)
    if (renderTimerRef.current) {
      clearTimeout(renderTimerRef.current);
    }
    const html = marked.parse(rawTextRef.current, {
      breaks: true,
      gfm: true,
    });
    setDisplayHtml(html as string);
  }, []);
 
  return { displayHtml, appendChunk, finalize };
};

The 50ms debounce is critical. Parsing Markdown on every chunk causes code blocks to render in broken states mid-stream. A 50ms buffer ensures most code blocks arrive complete before parsing runs.

For even more robustness, detect opening triple backticks and delay parsing until the block closes:

const isInsideCodeBlock = (text: string): boolean => {
  const matches = text.match(/```/g);
  return matches !== null && matches.length % 2 === 1;
};
 
// Inside appendChunk
renderTimerRef.current = setTimeout(() => {
  if (isInsideCodeBlock(rawTextRef.current)) {
    // Inside code block — defer parsing
    return;
  }
  // Normal parse
}, 50);

Rate Limiting and Cost Control

Streaming API calls share the same rate limits as standard calls. But when users send follow-up questions before the previous stream finishes, you can hit unexpected API call volumes:

class StreamManager {
  private activeController: AbortController | null = null;
 
  async startStream(prompt: string, onChunk: (text: string) => void) {
    // Cancel any existing stream
    if (this.activeController) {
      this.activeController.abort();
    }
 
    this.activeController = new AbortController();
    const { signal } = this.activeController;
 
    try {
      const stream = await model.generateContentStream(prompt, {
        signal,
      });
 
      for await (const chunk of stream.stream) {
        if (signal.aborted) break;
        const text = chunk.text();
        if (text) onChunk(text);
      }
    } catch (error) {
      if (error instanceof Error && error.name === 'AbortError') {
        // User sent a new question — normal cancellation
        return;
      }
      throw error;
    } finally {
      this.activeController = null;
    }
  }
}

Using AbortController to cancel the previous stream prevents wasted tokens and rate limit violations. This is the same mechanism behind ChatGPT's "Stop generating" button.

Your Next Move

The first priorities for production streaming are error recovery and buffering. Rendering optimizations can be layered on later, but shipping without disconnection handling will erode user trust in ways that are hard to reverse.

Mastering Gemini API Streaming Responses — Chunk Processing, Error Recovery, and UX Optimization