A Gemini stream drops halfway — restart it, or have the model continue?

Thinking tokens are paid twice on continuation

There is a term missing from that formula. On models with thinking enabled, a continuation request reasons again from scratch.

Continuation resumes the output, not the reasoning that produced it. With a generous thinkingBudget, regenerated thinking tokens routinely exceed the R × (Co − Ci) you were trying to save.

Because of this, I skip continuation entirely on paths where thinking runs deep. The cleaner fix turned out to be architectural: work that needs heavy reasoning does not belong on a stream a user is waiting on.

Gate on received tokens, not on a percentage

"Continue if we got 70% of the way there" is easy to write and frequently wrong. O is not known until generation finishes, so the denominator is a guess.

Drop the denominator. Decide on R alone — the output tokens actually in hand.

That 200 came out of my own operation and rests on feel, not on the cost model. Below it, a restart finishes quickly enough that rebuilding cleanly beats stitching. Your average output length will move that number, so please do not adopt it unexamined.

Structured-output paths go to restart unconditionally because a truncated JSON body, handed back for continuation, comes home with mismatched braces often enough to matter.

Checkpoint at sentence boundaries

You cannot hand back R cut at an arbitrary byte. Text severed mid-word makes the model try to complete the fragment, and the continuation opens strangely.

Rewind the buffer to the last sentence terminator before saving it. The Kotlin receiver ended up like this.

private val SENTENCE_END = Regex("[.!?。．！？\\n]")
 
class StreamCheckpoint {
    private val buffer = StringBuilder()
 
    /** Accumulate a chunk. Rendering it to the UI immediately is fine. */
    fun append(chunk: String) {
        buffer.append(chunk)
    }
 
    /**
     * Produce text safe to hand to a continuation request.
     * Truncates at the last sentence terminator and discards the partial sentence.
     * Returns null when no terminator has arrived yet (meaning: restart).
     */
    fun committedText(): String? {
        val text = buffer.toString()
        val lastEnd = SENTENCE_END.findAll(text).lastOrNull()?.range?.last ?: return null
        val committed = text.substring(0, lastEnd + 1).trimEnd()
        return committed.ifEmpty { null }
    }
 
    /** Rough token estimate, good enough for the gate. */
    fun approxTokens(): Int = (committedText()?.length ?: 0) / 4
}

Use approxTokens for the gate and reach for countTokens only when you truly need precision. Calling it on every disconnect adds a round trip to the very path that is already struggling.

Discarding the partial sentence looks wasteful. It is a few dozen tokens, rewritten by the continuation, in exchange for a seam that does not visibly break.

Send the partial output as a model turn

When you ask for the rest, resist embedding the partial text into the prompt as a quoted string. Push it into contents as a model turn and follow it with a short instruction.

from google import genai
from google.genai import types
 
client = genai.Client(api_key="YOUR_API_KEY")
 
def continue_generation(model, system_instruction, user_prompt, committed_text):
    """Replay the partial output as a model turn and ask only for the rest."""
    contents = [
        types.Content(role="user", parts=[types.Part(text=user_prompt)]),
        types.Content(role="model", parts=[types.Part(text=committed_text)]),
        types.Content(role="user", parts=[types.Part(
            text="Continue directly from the text above. "
                 "Do not restate, summarise, or preface it."
        )]),
    ]
    return client.models.generate_content_stream(
        model=model,
        contents=contents,
        config=types.GenerateContentConfig(
            system_instruction=system_instruction,
            temperature=0.4,
        ),
    )

Pass the same system_instruction you used on the first call. Omit it and the voice shifts, which is exactly what makes a seam visible to a reader.

The slightly lower temperature keeps the continuation from wandering. Finishing a piece needs less latitude than starting one.

Three ways the seam breaks

After shipping continuation, these were the defects that remained.

Restatement. Even with an explicit instruction, the model sometimes echoes the last sentence before moving on. Compare the first continuation chunk against the final ~40 characters of committed_text and drop the overlap.

Tonal steps. Continuing mid-paragraph reads slightly disjointed. I tried rewinding to paragraph boundaries instead, but the discarded volume grew too fast. Sentence boundaries, and living with the step, won.

Renumbered lists. A drop inside an ordered list often restarts numbering at 1. When committed_text ends on a numbered line, I append "the next number is N" to the instruction.

Each of these is a tax the continuation path pays. Handle all three and what remains in your pocket is R × (Co − Ci) — which is precisely why the threshold should not be set low.

Assume the result arrives twice

Continuation increases the odds of committing the same text twice: you declare a drop, fire the continuation, and the original stream's tail arrives late.

Give the write path an idempotency key and let it silently discard the loser. Issue the key per request and carry the same key into the continuation, because a continuation is not a new generation — it is the rest of one.

Where I landed, and what I am still unsure about

The current gate: structured output always restarts; free text continues above 200 received tokens; a second drop falls back to non-streaming; paths with deep thinking never continue.

The threshold is what still bothers me. The cost model only tells me "bigger R is better", so 200 comes from weighing seam-handling complexity against perceived wait. I suspect there is a better way to choose it.

Start by logging received token counts alongside your disconnect events. Once you can see the distribution of R, plugging it into the formula answers the question for your own setup. If your drops cluster around 100 tokens, the honest conclusion is that none of this is worth building — and knowing that is worth the logging.