You've built a chatbot or conversational assistant with the Gemini API, and it works great for the first few turns. Then, as the conversation grows, you hit something like this:
google.api_core.exceptions.InvalidArgument: 400
Request contains too many tokens. Please reduce the input size.
Or maybe you don't see an error at all, but responses start getting noticeably slower around turn 10, or the model suddenly "forgets" what you told it at the start of the conversation.
All of these trace back to the same root cause: how you're managing chat history. This article walks through three common multi-turn conversation problems and gives you practical, code-backed solutions for each.
Why Does a Long Conversation Cause Problems?
The key thing to understand: Gemini API's ChatSession sends the entire conversation history with every message.
import google.generativeai as genai
genai.configure(api_key="YOUR_GEMINI_API_KEY")
model = genai.GenerativeModel("gemini-2.5-flash")
chat = model.start_chat()
# Turn 1: Only sends "Hello"
response = chat.send_message("Hello")
# Turn 10: Sends all 9 previous exchanges PLUS your new message
# Turn 100: Sends all 99 previous exchanges...costs and latency grow linearlyEvery message accumulates in chat.history, and every subsequent request carries that growing weight. Light casual chat rarely hits a wall, but conversations involving technical explanations, long documents, or detailed context can exhaust your token budget surprisingly fast.
Problem 1: Token Overflow Error Mid-Conversation
What You See
A 400 Request contains too many tokens error appears out of nowhere. The previous exchange worked fine.
Fix: Check Token Count Before Sending
Adding a pre-send token check lets you trim history before hitting the limit rather than crashing into it.
import google.generativeai as genai
genai.configure(api_key="YOUR_GEMINI_API_KEY")
model = genai.GenerativeModel("gemini-2.5-flash")
def safe_send_message(chat, message, max_tokens=400000):
"""Check token count before sending a message."""
token_info = model.count_tokens(chat.history)
current_tokens = token_info.total_tokens
print(f"Current token count: {current_tokens:,}")
if current_tokens > max_tokens:
print("⚠️ Approaching token limit. Trimming old history...")
trim_history(chat, max_tokens)
return chat.send_message(message)
def trim_history(chat, max_tokens):
"""Remove old message pairs until we're under the token limit."""
while (model.count_tokens(chat.history).total_tokens > max_tokens
and len(chat.history) >= 2):
# Always remove in pairs: one user message + one model response
# Removing just one breaks the history structure and causes errors
chat.history.pop(0)
chat.history.pop(0)
# Usage
chat = model.start_chat()
for turn in range(50):
response = safe_send_message(
chat, f"Turn {turn + 1}: Tell me something about the Gemini API"
)
print(f"Turn {turn + 1}: {response.text[:60]}...")The critical detail here is removing history entries in pairs. The history alternates between user and model messages, so deleting just one entry corrupts the structure and causes a different error.
For a deeper look at this error, see Fixing Gemini API Token Limit Exceeded Errors.
Problem 2: The Model Loses Context After History Trimming
What You See
You implemented trim_history() to prevent overflow, but now the model forgets information you shared at the start of the conversation — the user's name, their goal, or the specific constraints you defined.
This is the fundamental limitation of simple deletion: the "important early context" is exactly what gets trimmed first.
Fix A: Pin Critical Information in system_instruction
Information that doesn't change during a conversation — user identity, preferences, language settings, persona — belongs in system_instruction, not in the chat history. It persists no matter how aggressively you trim.
import google.generativeai as genai
genai.configure(api_key="YOUR_GEMINI_API_KEY")
# Stable context goes in system_instruction, not in the chat
system_instruction = """
You are a helpful programming assistant.
User profile:
- Name: Alex
- Background: Software engineer with 3 years of Python experience
- Goal: Building a chatbot using the Gemini API
Always respond in English with explanations that are clear for intermediate developers.
"""
model = genai.GenerativeModel(
"gemini-2.5-flash",
system_instruction=system_instruction
)
chat = model.start_chat()
# Even after heavy history trimming, the system instruction stays intact
response = chat.send_message("What's my name and what am I working on?")
print(response.text)
# → "Your name is Alex, and you're building a chatbot with the Gemini API..."User names, goals, language preferences, and AI persona all belong in system_instruction. If it doesn't change during the conversation, it shouldn't live in the history.
Fix B: Compress Old History with Summarization
When the context does change and evolve over the conversation, the summarization approach preserves important information while dramatically reducing token count.
import google.generativeai as genai
genai.configure(api_key="YOUR_GEMINI_API_KEY")
model = genai.GenerativeModel("gemini-2.5-flash")
def summarize_old_history(history):
"""Summarize a chunk of history using a cost-efficient model."""
conversation_text = "\n".join([
f"{'User' if msg.role == 'user' else 'AI'}: {msg.parts[0].text}"
for msg in history
])
# Using a lightweight model for summarization keeps costs low
summary_model = genai.GenerativeModel("gemini-2.5-flash")
response = summary_model.generate_content(
f"Summarize this conversation in 3-5 sentences. "
f"Include any important facts, decisions, or context about the user:\n\n{conversation_text}"
)
return response.text
def compress_history(chat, keep_last_n=4):
"""Replace old history with a summary, keeping the most recent N turns."""
if len(chat.history) <= keep_last_n * 2:
return # Not enough history to compress yet
old_history = chat.history[:-keep_last_n * 2]
recent_history = chat.history[-keep_last_n * 2:]
summary = summarize_old_history(old_history)
print(f"📝 Compressed history: {summary[:80]}...")
# Replace old history with summary + recent conversation
chat.history = [
genai.protos.Content(role="user", parts=[
genai.protos.Part(text=f"[Summary of earlier conversation]\n{summary}")
]),
genai.protos.Content(role="model", parts=[
genai.protos.Part(text="Got it. I'll continue with that context in mind.")
])
] + recent_history
# Usage
chat = model.start_chat()
for i in range(20):
response = chat.send_message(f"Question {i + 1}: Share a Gemini API best practice")
print(f"Turn {i + 1}: {response.text[:50]}...")
# Compress every 8 turns
if (i + 1) % 8 == 0:
compress_history(chat, keep_last_n=4)
tokens = model.count_tokens(chat.history).total_tokens
print(f"✅ Token count after compression: {tokens:,}")Problem 3: Conversations Bleed Between Users
What You See
In a multi-user application, users start seeing each other's conversation history, or one user's context infects another's session.
What's Happening
A single ChatSession object is being shared across all requests:
# ❌ Wrong: One chat instance shared by all users
chat = model.start_chat() # Created once at startup
@app.route("/chat", methods=["POST"])
def handle_chat():
message = request.json["message"]
response = chat.send_message(message) # All users write to the same history
return {"response": response.text}Fix: Isolate Sessions Per User
import google.generativeai as genai
from flask import Flask, request, jsonify
genai.configure(api_key="YOUR_GEMINI_API_KEY")
model = genai.GenerativeModel("gemini-2.5-flash")
# Key each session by user ID
# For production: persist to Redis or a database
user_sessions: dict[str, genai.ChatSession] = {}
def get_or_create_chat(user_id: str) -> genai.ChatSession:
"""Return the ChatSession for this user, creating one if needed."""
if user_id not in user_sessions:
user_sessions[user_id] = model.start_chat()
print(f"✅ New session created for: {user_id}")
return user_sessions[user_id]
app = Flask(__name__)
@app.route("/chat", methods=["POST"])
def handle_chat():
user_id = request.json.get("user_id")
message = request.json.get("message")
if not user_id or not message:
return jsonify({"error": "user_id and message are required"}), 400
chat = get_or_create_chat(user_id)
# Auto-compress if token count is getting high
token_count = model.count_tokens(chat.history).total_tokens
if token_count > 300000:
compress_history(chat, keep_last_n=6)
response = chat.send_message(message)
return jsonify({
"response": response.text,
"session_turns": len(chat.history) // 2,
"current_tokens": model.count_tokens(chat.history).total_tokens
})
if __name__ == "__main__":
app.run(debug=True)In-memory storage works for development but disappears on restart. For production, serialize history to JSON and store it in Redis using a key like user:{id}:history.
Persisting Chat History Across Sessions
For development workflows where you want conversation to survive app restarts, JSON serialization works well:
import json
import google.generativeai as genai
genai.configure(api_key="YOUR_GEMINI_API_KEY")
model = genai.GenerativeModel("gemini-2.5-flash")
def save_history(chat: genai.ChatSession, filepath: str) -> None:
"""Save chat history to a JSON file."""
history_data = [
{
"role": msg.role,
"parts": [p.text for p in msg.parts if hasattr(p, "text")]
}
for msg in chat.history
]
with open(filepath, "w", encoding="utf-8") as f:
json.dump(history_data, f, indent=2)
print(f"💾 Saved {len(history_data)} messages")
def load_history(filepath: str) -> genai.ChatSession:
"""Restore a chat session from a saved JSON file."""
try:
with open(filepath, "r", encoding="utf-8") as f:
history_data = json.load(f)
history = [
genai.protos.Content(
role=msg["role"],
parts=[genai.protos.Part(text=p) for p in msg["parts"]]
)
for msg in history_data
]
chat = model.start_chat(history=history)
print(f"✅ Restored {len(history)} messages")
return chat
except FileNotFoundError:
print("📝 No history found. Starting fresh.")
return model.start_chat()
# Usage
HISTORY_FILE = "chat_history.json"
chat = load_history(HISTORY_FILE)
response = chat.send_message("Let's continue where we left off.")
print(response.text)
# Save on exit
save_history(chat, HISTORY_FILE)What to Do Next
Here's the short version of what we covered:
- Token overflow → Check with
count_tokens()before sending, trim old message pairs when you're close to the limit - Context loss → Pin stable information in
system_instruction, compress evolving context with summarization - Session bleeding → Create one
ChatSessionper user ID, auto-compress as token counts grow
If you're using the same system prompt repeatedly across many conversations, Context Caching for Cost Optimization is worth exploring — it lets you cache the static parts and avoid paying for them on every request.
And if scaling up your chatbot leads to rate limit issues, Handling Gemini API Quota and 429 Errors covers the patterns you'll need.