◈ API / SDK/2026-07-17Advanced

A Gemini stream drops halfway — restart it, or have the model continue?

Most apps silently restart a dropped stream. Here is the arithmetic behind continuing from the partial output instead, and where to put the threshold.

Gemini API¹⁹¹ streaming²⁸ cost design⁵ idempotency⁵ mobile⁶

✦ Premium Article

My train pulled into a station and the text my app was generating stopped mid-sentence. The app quietly started over, and the same opening paragraph scrolled past a second time.

What stuck with me was not the wait. It was that the 600 tokens of prose that had just vanished were almost certainly billed. The user never saw a word of it, and a second generation was already underway.

Google's Southeast Asia report put roughly three quarters of Gemini app requests on mobile. Flaky connections are not an edge case you handle later; they are the normal operating condition. As an indie developer shipping Gemini into mobile apps, I have stopped treating a dropped stream as an anomaly.

Why streams drop is a separate investigation. This piece is about the decision you make afterwards.

You cannot tell how much a dropped stream cost you

The usageMetadata block on streamGenerateContent arrives near the end of the chunk sequence. If you receive the stream to completion, you get exact input, output, and thinking token counts.

Which means that for the attempt that died, you get nothing. The client has no way to confirm how far generation had progressed when the connection dropped.

And what happens server-side when a client disconnects — whether generation stops immediately, whether billing stops with it — is not spelled out in the public documentation.

Assuming "probably not billed" quietly corrupts your design. I work on the assumption that every chunk I received was paid for. That is not pessimism so much as refusing to round an unknown in my own favour.

Situation	usageMetadata	Actual billing
Stream completes	Available	Known exactly
Connection drops	Never arrives	Unknown — assume the worst
Client cancels	Never arrives	Unknown (same)

If you cannot measure the spend, at least count the events. Logging each disconnect with the number of characters already received costs nothing and pays off later.

Restarting and continuing have different cost shapes

Let P be the prompt tokens, O the output tokens needed to finish, and R the output tokens you received before the drop. Let Ci be the input rate and Co the output rate.

The dead attempt (Ci×P + Co×R) is spent either way. The difference shows up in what comes next.

Approach	Second input	Second output	What you throw away
Restart	P	O	The R tokens you already have
Continue	P + R	O − R	A clean seam

Subtract one from the other:

restart − continue
 = (Ci×P + Co×O) − (Ci×(P+R) + Co×(O−R))
 = Co×R − Ci×R
 = R × (Co − Ci)

Gemini prices output above input, so Co − Ci is positive and continuing is always nominally cheaper.

Read the formula more carefully, though. The saving scales with R. At R = 60 tokens you save 60 × (Co − Ci), which will not repay the code you are about to write.

Deciding "always continue" on cost alone misreads this result. Continuation earns its keep only when R is large — when a long piece of prose died near the end.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦You can replace blanket retries with a rule that decides per drop whether to reuse the text you already received

✦You can plug your own pricing into R × (output rate − input rate) and see whether continuation is worth building at all

✦You can fix the three seam defects — restatement, tonal steps, renumbered lists — before they reach users

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

Thinking tokens are paid twice on continuation

There is a term missing from that formula. On models with thinking enabled, a continuation request reasons again from scratch.

Continuation resumes the output, not the reasoning that produced it. With a generous thinkingBudget, regenerated thinking tokens routinely exceed the R × (Co − Ci) you were trying to save.

Because of this, I skip continuation entirely on paths where thinking runs deep. The cleaner fix turned out to be architectural: work that needs heavy reasoning does not belong on a stream a user is waiting on.

Gate on received tokens, not on a percentage

"Continue if we got 70% of the way there" is easy to write and frequently wrong. O is not known until generation finishes, so the denominator is a guess.

Drop the denominator. Decide on R alone — the output tokens actually in hand.

Condition	Choice	Reason
responseSchema is set	Restart	Partial JSON cannot be closed by continuation
R below threshold (~200 tokens)	Restart	Saving does not justify the seam
R at or above threshold, free text	Continue	Too much to throw away
Second drop on the same request	Fall back to non-streaming	The connection is not holding

That 200 came out of my own operation and rests on feel, not on the cost model. Below it, a restart finishes quickly enough that rebuilding cleanly beats stitching. Your average output length will move that number, so please do not adopt it unexamined.

Structured-output paths go to restart unconditionally because a truncated JSON body, handed back for continuation, comes home with mismatched braces often enough to matter.

Checkpoint at sentence boundaries

You cannot hand back R cut at an arbitrary byte. Text severed mid-word makes the model try to complete the fragment, and the continuation opens strangely.

Rewind the buffer to the last sentence terminator before saving it. The Kotlin receiver ended up like this.

private val SENTENCE_END = Regex("[.!?。．！？\\n]")
 
class StreamCheckpoint {
    private val buffer = StringBuilder()
 
    /** Accumulate a chunk. Rendering it to the UI immediately is fine. */
    fun append(chunk: String) {
        buffer.append(chunk)
    }
 
    /**
     * Produce text safe to hand to a continuation request.
     * Truncates at the last sentence terminator and discards the partial sentence.
     * Returns null when no terminator has arrived yet (meaning: restart).
     */
    fun committedText(): String? {
        val text = buffer.toString()
        val lastEnd = SENTENCE_END.findAll(text).lastOrNull()?.range?.last ?: return null
        val committed = text.substring(0, lastEnd + 1).trimEnd()
        return committed.ifEmpty { null }
    }
 
    /** Rough token estimate, good enough for the gate. */
    fun approxTokens(): Int = (committedText()?.length ?: 0) / 4
}

Use approxTokens for the gate and reach for countTokens only when you truly need precision. Calling it on every disconnect adds a round trip to the very path that is already struggling.

Discarding the partial sentence looks wasteful. It is a few dozen tokens, rewritten by the continuation, in exchange for a seam that does not visibly break.

Send the partial output as a model turn

When you ask for the rest, resist embedding the partial text into the prompt as a quoted string. Push it into contents as a model turn and follow it with a short instruction.

from google import genai
from google.genai import types
 
client = genai.Client(api_key="YOUR_API_KEY")
 
def continue_generation(model, system_instruction, user_prompt, committed_text):
    """Replay the partial output as a model turn and ask only for the rest."""
    contents = [
        types.Content(role="user", parts=[types.Part(text=user_prompt)]),
        types.Content(role="model", parts=[types.Part(text=committed_text)]),
        types.Content(role="user", parts=[types.Part(
            text="Continue directly from the text above. "
                 "Do not restate, summarise, or preface it."
        )]),
    ]
    return client.models.generate_content_stream(
        model=model,
        contents=contents,
        config=types.GenerateContentConfig(
            system_instruction=system_instruction,
            temperature=0.4,
        ),
    )

Pass the same system_instruction you used on the first call. Omit it and the voice shifts, which is exactly what makes a seam visible to a reader.

The slightly lower temperature keeps the continuation from wandering. Finishing a piece needs less latitude than starting one.

Three ways the seam breaks

After shipping continuation, these were the defects that remained.

Restatement. Even with an explicit instruction, the model sometimes echoes the last sentence before moving on. Compare the first continuation chunk against the final ~40 characters of committed_text and drop the overlap.

Tonal steps. Continuing mid-paragraph reads slightly disjointed. I tried rewinding to paragraph boundaries instead, but the discarded volume grew too fast. Sentence boundaries, and living with the step, won.

Renumbered lists. A drop inside an ordered list often restarts numbering at 1. When committed_text ends on a numbered line, I append "the next number is N" to the instruction.

Each of these is a tax the continuation path pays. Handle all three and what remains in your pocket is R × (Co − Ci) — which is precisely why the threshold should not be set low.

Assume the result arrives twice

Continuation increases the odds of committing the same text twice: you declare a drop, fire the continuation, and the original stream's tail arrives late.

Give the write path an idempotency key and let it silently discard the loser. Issue the key per request and carry the same key into the continuation, because a continuation is not a new generation — it is the rest of one.

Where I landed, and what I am still unsure about

The current gate: structured output always restarts; free text continues above 200 received tokens; a second drop falls back to non-streaming; paths with deep thinking never continue.

The threshold is what still bothers me. The cost model only tells me "bigger R is better", so 200 comes from weighing seam-handling complexity against perceived wait. I suspect there is a better way to choose it.

Start by logging received token counts alongside your disconnect events. Once you can see the distribution of R, plugging it into the formula answers the question for your own setup. If your drops cluster around 100 tokens, the honest conclusion is that none of this is worth building — and knowing that is worth the logging.

Thank You for Reading

Gemini Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.