When Gemini API Streaming Cuts Off Mid-Response in Production: The Diagnosis Order I Run

In early May, the Gemini API streaming I had wired into a wallpaper-app admin dashboard started cutting off mid-token on certain requests. The stream loop would exit cleanly, but the UI showed an unfinished response and a hanging ellipsis. I burned a full day on whether to blame the network, the SDK, or the server, so I have written down the diagnosis order I now run on this kind of bug.

I am Masaki Hirokawa, an indie developer running wallpaper apps since 2014 (over 50 million downloads). I use the Gemini API for article generation and as an assistant inside internal dashboards. The story below comes from one of those dashboards.

Three facts to confirm before diagnosing

Before walking through suspects, make sure the symptom is actually a streaming cutoff. Skipping this step is how you mistake a JSON parse failure for a network problem. Confirm these three within five minutes:

The finish_reason of the last chunk received by your SDK iterator.
Whether the cumulative token count from the server matches the characters that actually reached the UI.
Whether the symptom is consistent (every request) or sporadic (some requests).

In my case the last chunk had finish_reason = "OTHER", the server-reported cumulative tokens were roughly double what the UI displayed, and the symptom was sporadic. That immediately framed it as "server thinks it finished, but the client stopped receiving."

Diagnosis order

Here is the order I now apply, starting with the cheapest suspect:

1. Client-side timeout

Check the event-loop timeout configuration first. Mine had aiohttp.ClientTimeout(total=30) on the session, which silently killed long responses. For streaming you want sock_read configured rather than total, or a generous streaming-specific value.

import aiohttp
# For streaming, drop total and only bound the read interval
timeout = aiohttp.ClientTimeout(total=None, sock_read=60)
async with aiohttp.ClientSession(timeout=timeout) as s:
    ...

This step alone removed about 60% of the sporadic cutoffs.

2. Proxy / CDN buffering

Next, suspect the middle of the network. One of my paths routes through a Cloudflare Worker, and I hit cases where long idle gaps inside a response cause the connection to drop. The Gemini SDK does send keepalive packets, but if the prompt produces long stretches with no token output (think long JSON formatting), the intermediary CDN can drop the connection anyway.

My mitigation is to instruct prompts that are likely to produce long, mostly-idle outputs to "emit progressively" so that at least one token flows every 5 seconds. That helps but is not bulletproof, which is why step 3 exists.

3. SDK-side reconnect strategy

The Gemini SDK does not reconnect on a mid-stream drop. To get reconnection you need your own wrapper.

async def stream_with_retry(prompt: str, max_retry: int = 2):
    accumulated = ""
    for attempt in range(max_retry + 1):
        try:
            async for chunk in client.aio.models.generate_content_stream(
                model="gemini-2.5-flash", contents=accumulated + prompt if accumulated else prompt,
            ):
                accumulated += chunk.text
                yield chunk.text
            return
        except Exception:
            if attempt == max_retry:
                raise
            await asyncio.sleep(0.5)

This wrapper effectively asks for the continuation from where the stream broke, and the UI keeps consuming chunks normally. After dropping this in, the sporadic cutoffs effectively vanished from my dashboard.

4. Safety-filter truncation

Last, check the safety filters. If finish_reason is SAFETY or RECITATION, no amount of reconnection will help. In my case wallpaper-app review summarization occasionally hits RECITATION, and I work around it by instructing the prompt to avoid quote-style output.

A short incident-record template

Every time this class of issue hits, I leave a short record:

Timestamp / reproduction rate (M out of N tries)
finish_reason
Measured input/output token counts
Network path (direct / proxy / Worker / etc.)
Mitigation applied

After about five records, the highest-leverage mitigation for your specific environment will become obvious. For me the sock_read change was the single most cost-effective fix; the SDK wrapper was insurance.

What to try first if you are stuck right now

If you are debugging a cutoff right now, start by printing finish_reason on every chunk. That single value narrows the suspect area to about a third of the original space.

Job stoppages are the kind of issue that quietly costs revenue. I hope this helps you fill out your own diagnosis toolkit.