◉GEMINI LABJP
●OMNI 1.2 — Gemini Omni 1.2 Flash is generally available (11/5) with up to 15-second video and conversational editing; 1.1 Flash is routed to 1.2 on 12/3●COMPUTER — Computer Use is now generally available in the Interactions and GenerateContent APIs (10/30)●LIVE — Live and TTS preview models shut down as early as 11/17 (7 days left). Move to gemini-3.8-live and the 3.8 TTS models●CLI — Gemini API CLI (gemini-api) is out in beta (10/27): a zero-dependency client for agents and CI●2.5 PRO — gemini-2.5-pro and 2.5-flash are routed automatically on 12/1 (21 days left), to 3.8 Flash and 3.5 Flash-Lite●NEW — Stop Re-Reading Every Crash Report: Gre…●OMNI 1.2 — Gemini Omni 1.2 Flash is generally available (11/5) with up to 15-second video and conversational editing; 1.1 Flash is routed to 1.2 on 12/3●COMPUTER — Computer Use is now generally available in the Interactions and GenerateContent APIs (10/30)●LIVE — Live and TTS preview models shut down as early as 11/17 (7 days left). Move to gemini-3.8-live and the 3.8 TTS models●CLI — Gemini API CLI (gemini-api) is out in beta (10/27): a zero-dependency client for agents and CI●2.5 PRO — gemini-2.5-pro and 2.5-flash are routed automatically on 12/1 (21 days left), to 3.8 Flash and 3.5 Flash-Lite●NEW — Stop Re-Reading Every Crash Report: Gre…
Articles/API / SDK
◈ API / SDK/2026-05-20Intermediate

Gemini API Streaming Works Locally but Buffers in Production — Fixing Cloud Run, Vercel, and Cloudflare

Streaming responses flow token-by-token in local dev, then arrive as one big blob in production. A walkthrough of the five most common causes — Cloud Run timeouts, Vercel runtime mismatch, Cloudflare Workers proxying, server-side text() pitfalls, and client-side decoding — with the fixes I use on my own sites.

gemini-api291streaming28troubleshooting98cloud-run6vercelcloudflare4

"On my laptop the Gemini response streams one character at a time. After deploying, the user just stares at a spinner for thirty seconds before the entire answer drops in at once." This is the single issue I have debugged more often than any other. The API itself is fine; streamGenerateContent is doing its job. Something between the model and the browser is buffering the chunks, and from the user's perspective it might as well be a frozen tab.

This post walks through the five most common reasons production-only buffering happens and the concrete fix for each platform. I keep this checklist in front of me whenever I ship a new endpoint that talks to Gemini, because forgetting any one of the steps reliably breaks the experience.

First, prove that the chunks really are being buffered

In my experience, most "streaming is broken" reports that only appear in production turn out to be a layer between the server and the browser holding chunks back. Before reaching for code changes, isolate whether the server is emitting chunks gradually and whether the client is reading them gradually. Two independent questions.

The most reliable test is curl --no-buffer. From a shell, hit the production endpoint directly and watch whether bytes appear progressively.

curl --no-buffer -N \
  -H "Content-Type: application/json" \
  -X POST "https://your-app.example.com/api/chat" \
  -d '{"prompt": "Long streaming response, please."}' \
  2>&1 | tee /tmp/stream.log

--no-buffer disables curl's own stdout buffering and -N makes curl handle chunked transfer encoding without waiting. If the terminal stays silent for thirty seconds and then dumps the entire payload, the network path is buffering. If text trickles in steadily, the server is healthy and the bug is in the browser code — probably fetch ReadableStream handling.

In Chrome DevTools, open the Network panel and look at the Timing breakdown for the request. A long Content Download phase means streaming is working. A long Waiting (TTFB) with a near-instant Content Download means the server is generating the full response before flushing anything.

Cause 1: the server collected the response before returning it

This is the most common bug in my code reviews. Calling generateContentStream() and then doing JSON.stringify(result) or await response.text() somewhere in the middle defeats streaming completely. In Next.js Route Handlers, Hono, Elysia, or any modern framework, the goal is to hand a ReadableStream straight to the Response constructor.

// app/api/chat/route.ts (Next.js App Router, Node runtime)
import { GoogleGenAI } from "@google/genai";
 
export const runtime = "nodejs"; // do not switch to edge here without reading Cause 3
export const dynamic = "force-dynamic";
 
export async function POST(req: Request) {
  const { prompt } = await req.json();
  const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY! });
 
  const stream = new ReadableStream({
    async start(controller) {
      const encoder = new TextEncoder();
      try {
        const result = await ai.models.generateContentStream({
          model: "gemini-2.5-flash", // swap in whichever model is currently available
          contents: prompt,
        });
        for await (const chunk of result) {
          const text = chunk.text;
          if (text) controller.enqueue(encoder.encode(text));
        }
      } catch (err) {
        controller.error(err);
      } finally {
        controller.close();
      }
    },
  });
 
  return new Response(stream, {
    headers: {
      "Content-Type": "text/plain; charset=utf-8",
      "Cache-Control": "no-cache, no-transform",
      "X-Accel-Buffering": "no",
      "Transfer-Encoding": "chunked",
    },
  });
}

The two headers worth memorising are Cache-Control: no-transform and X-Accel-Buffering: no. The latter is a hint for Nginx-family reverse proxies. It matters most if you run Nginx yourself; managed paths may ignore it, so I treat it as insurance rather than a fix.

If your sample code still uses @google/generative-ai, migrate to @google/genai first. The stream handling changes (no result.stream, and chunk.text is a property), and it is easy to get lost in the difference.

Cause 2: Cloud Run timeouts and HTTP/2

Cloud Run handles streaming fine on its own, but I have tripped on two things.

The first is the request timeout. The default is 300 seconds, but if an older setting left it at 60, a long Gemini 2.5 Pro response gets cut mid-stream and the client sees a 504. Set it explicitly so you are not guessing:

gcloud run services update YOUR_SERVICE --timeout=300

The second is HTTP/2. --use-http2 switches the service over, but it assumes your container speaks h2c. A plain Node HTTP server may stop answering after the switch, so run a curl check right after flipping it.

When I put a category-suggestion endpoint for my wallpaper apps on Cloud Run, chunks arrived in clumps of a few seconds. I assumed HTTP/1.1 was the cause and enabled --use-http2. What actually helped was flushing explicitly on the server after each piece of output. A setting change that feels like a fix is not a fix until I have re-measured with curl --no-buffer. I try not to skip that line anymore.

Cause 3: Vercel Edge runtime vs Node runtime

On Vercel, the runtime directive on a Route Handler changes everything about streaming behaviour.

SettingStreamingMax durationNotes
runtime = "edge"Native, stableVaries by plan (check the official Functions limits)Built on Web Streams; pairs well with fetch-based code
runtime = "nodejs"Works with force-dynamicSet by plan and maxDurationOmitting dynamic = "force-dynamic" causes static generation

The two failure modes I see most often: an Edge function running into its call-count or duration limits, and forgetting export const dynamic = "force-dynamic" on Node runtime, which silently turns the route into a build-time static export. In the latter case every user gets the same prebuilt response, instantly — it looks like streaming broke when in fact nothing is being generated at all.

To raise the maximum duration, set it per function in vercel.json (the ceiling depends on your plan):

{
  "functions": {
    "app/api/chat/route.ts": {
      "maxDuration": 300
    }
  }
}

Cause 4: Cloudflare Workers and Pages Functions proxying

The sites I run are on Cloudflare Workers, so I trip over this layer regularly. Workers handles ReadableStream natively, but the moment you slot in a caching middleware (something like cache-worker.js) it is easy to introduce a line that collapses the stream.

A correct relay looks like this:

// _workers/cache-worker.js (simplified)
export default {
  async fetch(request, env, ctx) {
    const url = new URL(request.url);
    if (url.pathname.startsWith("/api/chat")) {
      const upstream = await fetch(request);
      return new Response(upstream.body, {
        status: upstream.status,
        headers: {
          ...Object.fromEntries(upstream.headers),
          "Cache-Control": "no-cache, no-transform",
        },
      });
    }
    return cacheHandler(request, env, ctx);
  },
};

The key is passing upstream.body straight into the new Response. Any path that does response.clone() inside a Promise.all, or await response.text(), ends up waiting for the entire upstream response before forwarding it. The Worker becomes the bottleneck.

A related quirk: when the Worker's execution location and the origin are far apart (Smart Placement, for example), the round-trip delay can look like buffering. I measure first and decide whether to enable it afterwards.

Cause 5: client-side fetch and ReadableStream pitfalls

Even with a flawless server, browser code that waits for the full response before rendering produces the same UX as buffering. The fetch ReadableStream needs to be iterated chunk-by-chunk.

async function streamChat(prompt: string, onToken: (t: string) => void) {
  const res = await fetch("/api/chat", {
    method: "POST",
    body: JSON.stringify({ prompt }),
    headers: { "Content-Type": "application/json" },
  });
  if (!res.body) throw new Error("no body");
 
  const reader = res.body.getReader();
  const decoder = new TextDecoder();
  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    onToken(decoder.decode(value, { stream: true }));
  }
}

The detail that bites people is { stream: true } on the TextDecoder.decode call. Without it, multibyte characters (Japanese, emoji, anything outside basic ASCII) that span a chunk boundary get dropped or replaced. The visible result is "fifty characters appear every few seconds" — looks like buffering, is actually decoder behaviour.

In React, calling setState on every chunk can thrash painting on long outputs. I accumulate text in a useRef and flush it once per animation frame; heavy transforms like react-markdown run once at the end.

Pre-deploy checklist

These are the six items I run through before pushing any new Gemini endpoint to production.

  1. curl --no-buffer -N shows characters arriving progressively.
  2. Response headers include Cache-Control: no-cache, no-transform and X-Accel-Buffering: no.
  3. Cloud Run has --timeout set explicitly (and --use-http2 only together with a connectivity check).
  4. Vercel Node-runtime handlers explicitly set export const dynamic = "force-dynamic".
  5. Cloudflare Workers relays use upstream.body directly, without text() or clone().then(...).
  6. Client-side TextDecoder.decode(value, { stream: true }) is in place.

Since I started walking through this list in this order, the hours I lose to "streaming mysteriously dies in production" have shrunk a lot.

Closing thoughts

Production-only streaming failures are almost always a path problem — Cloud Run, Vercel, Cloudflare, Nginx, or a piece of code that accidentally collects the full response — rather than something Gemini itself is doing wrong. Start by isolating whether the server is emitting chunks at all (curl --no-buffer), then check the server implementation for hidden text() calls, then go through the platform-specific knobs in this post.

One more thing I have learned: when a settings change seems to fix it, I still can't tell which change did the work until I measure again.

If you have time for one concrete next step, point that curl --no-buffer command at your live endpoint. If you see silence followed by a single dump, one of the sections above will explain it.

I hope this shortens someone's debugging weekend.

Share

Thank You for Reading

Gemini Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.

  • ✦Copy-paste ready implementation code
  • ✦New advanced guides published daily
  • ✦$5/mo or $10 for lifetime access
View Membership →

If you found this article helpful, a small tip ($1.50) would mean a lot to us. Your support helps keep this site ad-free and covers server and hosting costs.

Related Articles

◈ API / SDK2026-10-19
When 404 and 429 Show Up in the Same Gemini API Log, Check the Model Name Before the Quota
A triage order for Gemini API logs that mix 404 NOT_FOUND and 429 RESOURCE_EXHAUSTED, plus a small Python script that separates a bad model name from a real quota limit without generating anything.
◈ API / SDK2026-10-16
The Day /models Returns 200 but generateContent Returns 403 — Decide Your Triage Order Before It Happens
A triage order for the case where listing models succeeds with 200 but generation fails with 403 permission_denied. Includes a small diagnostic script that checks key, project, and billing in turn, why you should not retry a 403, and how to assemble what support needs.
◈ API / SDK2026-05-17
Diagnosing INVALID_ARGUMENT Errors in Gemini API Batch Image Analysis
A batch of images to the Gemini API fails with INVALID_ARGUMENT (HTTP 400). This guide gives a triage table for MIME type, request size, contents structure and corrupt files, pre-flight checks rewritten for the google-genai SDK, and a rule for choosing inline data vs the File API (as of November 2026).