"On my laptop the Gemini response streams one character at a time. After deploying, the user just stares at a spinner for thirty seconds before the entire answer drops in at once." This is the single issue I have debugged more often than any other. The API itself is fine; streamGenerateContent is doing its job. Something between the model and the browser is buffering the chunks, and from the user's perspective it might as well be a frozen tab.
This post walks through the five most common reasons production-only buffering happens and the concrete fix for each platform. I keep this checklist in front of me whenever I ship a new endpoint that talks to Gemini, because forgetting any one of the steps reliably breaks the experience.
First, prove that the chunks really are being buffered
In my experience, most "streaming is broken" reports that only appear in production turn out to be a layer between the server and the browser holding chunks back. Before reaching for code changes, isolate whether the server is emitting chunks gradually and whether the client is reading them gradually. Two independent questions.
The most reliable test is curl --no-buffer. From a shell, hit the production endpoint directly and watch whether bytes appear progressively.
curl --no-buffer -N \
-H "Content-Type: application/json" \
-X POST "https://your-app.example.com/api/chat" \
-d '{"prompt": "Long streaming response, please."}' \
2>&1 | tee /tmp/stream.log--no-buffer disables curl's own stdout buffering and -N makes curl handle chunked transfer encoding without waiting. If the terminal stays silent for thirty seconds and then dumps the entire payload, the network path is buffering. If text trickles in steadily, the server is healthy and the bug is in the browser code — probably fetch ReadableStream handling.
In Chrome DevTools, open the Network panel and look at the Timing breakdown for the request. A long Content Download phase means streaming is working. A long Waiting (TTFB) with a near-instant Content Download means the server is generating the full response before flushing anything.
Cause 1: the server collected the response before returning it
This is the most common bug in my code reviews. Calling generateContentStream() and then doing JSON.stringify(result) or await response.text() somewhere in the middle defeats streaming completely. In Next.js Route Handlers, Hono, Elysia, or any modern framework, the goal is to hand a ReadableStream straight to the Response constructor.
// app/api/chat/route.ts (Next.js App Router, Node runtime)
import { GoogleGenAI } from "@google/genai";
export const runtime = "nodejs"; // do not switch to edge here without reading Cause 3
export const dynamic = "force-dynamic";
export async function POST(req: Request) {
const { prompt } = await req.json();
const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY! });
const stream = new ReadableStream({
async start(controller) {
const encoder = new TextEncoder();
try {
const result = await ai.models.generateContentStream({
model: "gemini-2.5-flash", // swap in whichever model is currently available
contents: prompt,
});
for await (const chunk of result) {
const text = chunk.text;
if (text) controller.enqueue(encoder.encode(text));
}
} catch (err) {
controller.error(err);
} finally {
controller.close();
}
},
});
return new Response(stream, {
headers: {
"Content-Type": "text/plain; charset=utf-8",
"Cache-Control": "no-cache, no-transform",
"X-Accel-Buffering": "no",
"Transfer-Encoding": "chunked",
},
});
}The two headers worth memorising are Cache-Control: no-transform and X-Accel-Buffering: no. The latter is a hint for Nginx-family reverse proxies. It matters most if you run Nginx yourself; managed paths may ignore it, so I treat it as insurance rather than a fix.
If your sample code still uses @google/generative-ai, migrate to @google/genai first. The stream handling changes (no result.stream, and chunk.text is a property), and it is easy to get lost in the difference.
Cause 2: Cloud Run timeouts and HTTP/2
Cloud Run handles streaming fine on its own, but I have tripped on two things.
The first is the request timeout. The default is 300 seconds, but if an older setting left it at 60, a long Gemini 2.5 Pro response gets cut mid-stream and the client sees a 504. Set it explicitly so you are not guessing:
gcloud run services update YOUR_SERVICE --timeout=300The second is HTTP/2. --use-http2 switches the service over, but it assumes your container speaks h2c. A plain Node HTTP server may stop answering after the switch, so run a curl check right after flipping it.
When I put a category-suggestion endpoint for my wallpaper apps on Cloud Run, chunks arrived in clumps of a few seconds. I assumed HTTP/1.1 was the cause and enabled --use-http2. What actually helped was flushing explicitly on the server after each piece of output. A setting change that feels like a fix is not a fix until I have re-measured with curl --no-buffer. I try not to skip that line anymore.
Cause 3: Vercel Edge runtime vs Node runtime
On Vercel, the runtime directive on a Route Handler changes everything about streaming behaviour.
| Setting | Streaming | Max duration | Notes |
|---|---|---|---|
runtime = "edge" | Native, stable | Varies by plan (check the official Functions limits) | Built on Web Streams; pairs well with fetch-based code |
runtime = "nodejs" | Works with force-dynamic | Set by plan and maxDuration | Omitting dynamic = "force-dynamic" causes static generation |
The two failure modes I see most often: an Edge function running into its call-count or duration limits, and forgetting export const dynamic = "force-dynamic" on Node runtime, which silently turns the route into a build-time static export. In the latter case every user gets the same prebuilt response, instantly — it looks like streaming broke when in fact nothing is being generated at all.
To raise the maximum duration, set it per function in vercel.json (the ceiling depends on your plan):
{
"functions": {
"app/api/chat/route.ts": {
"maxDuration": 300
}
}
}Cause 4: Cloudflare Workers and Pages Functions proxying
The sites I run are on Cloudflare Workers, so I trip over this layer regularly. Workers handles ReadableStream natively, but the moment you slot in a caching middleware (something like cache-worker.js) it is easy to introduce a line that collapses the stream.
A correct relay looks like this:
// _workers/cache-worker.js (simplified)
export default {
async fetch(request, env, ctx) {
const url = new URL(request.url);
if (url.pathname.startsWith("/api/chat")) {
const upstream = await fetch(request);
return new Response(upstream.body, {
status: upstream.status,
headers: {
...Object.fromEntries(upstream.headers),
"Cache-Control": "no-cache, no-transform",
},
});
}
return cacheHandler(request, env, ctx);
},
};The key is passing upstream.body straight into the new Response. Any path that does response.clone() inside a Promise.all, or await response.text(), ends up waiting for the entire upstream response before forwarding it. The Worker becomes the bottleneck.
A related quirk: when the Worker's execution location and the origin are far apart (Smart Placement, for example), the round-trip delay can look like buffering. I measure first and decide whether to enable it afterwards.
Cause 5: client-side fetch and ReadableStream pitfalls
Even with a flawless server, browser code that waits for the full response before rendering produces the same UX as buffering. The fetch ReadableStream needs to be iterated chunk-by-chunk.
async function streamChat(prompt: string, onToken: (t: string) => void) {
const res = await fetch("/api/chat", {
method: "POST",
body: JSON.stringify({ prompt }),
headers: { "Content-Type": "application/json" },
});
if (!res.body) throw new Error("no body");
const reader = res.body.getReader();
const decoder = new TextDecoder();
while (true) {
const { done, value } = await reader.read();
if (done) break;
onToken(decoder.decode(value, { stream: true }));
}
}The detail that bites people is { stream: true } on the TextDecoder.decode call. Without it, multibyte characters (Japanese, emoji, anything outside basic ASCII) that span a chunk boundary get dropped or replaced. The visible result is "fifty characters appear every few seconds" — looks like buffering, is actually decoder behaviour.
In React, calling setState on every chunk can thrash painting on long outputs. I accumulate text in a useRef and flush it once per animation frame; heavy transforms like react-markdown run once at the end.
Pre-deploy checklist
These are the six items I run through before pushing any new Gemini endpoint to production.
curl --no-buffer -Nshows characters arriving progressively.- Response headers include
Cache-Control: no-cache, no-transformandX-Accel-Buffering: no. - Cloud Run has
--timeoutset explicitly (and--use-http2only together with a connectivity check). - Vercel Node-runtime handlers explicitly set
export const dynamic = "force-dynamic". - Cloudflare Workers relays use
upstream.bodydirectly, withouttext()orclone().then(...). - Client-side
TextDecoder.decode(value, { stream: true })is in place.
Since I started walking through this list in this order, the hours I lose to "streaming mysteriously dies in production" have shrunk a lot.
Closing thoughts
Production-only streaming failures are almost always a path problem — Cloud Run, Vercel, Cloudflare, Nginx, or a piece of code that accidentally collects the full response — rather than something Gemini itself is doing wrong. Start by isolating whether the server is emitting chunks at all (curl --no-buffer), then check the server implementation for hidden text() calls, then go through the platform-specific knobs in this post.
One more thing I have learned: when a settings change seems to fix it, I still can't tell which change did the work until I measure again.
If you have time for one concrete next step, point that curl --no-buffer command at your live endpoint. If you see silence followed by a single dump, one of the sections above will explain it.
I hope this shortens someone's debugging weekend.