Growing a Customer Support Chatbot with Gemini API: An Implementation Notebook

Three-Layer System Prompt

A common early mistake is writing the system prompt as a single long paragraph. Six months later, when you want to tweak one thing, you can't find where you said it. I split mine into three layers.

The first layer is "Role and Responsibilities": who the bot is. The second is "Response Guidelines": tone and format. The third is "Escalation Conditions": when to hand off. With these layers separated, you can change tone without touching escalation, or tighten escalation rules without rewriting the persona.

This structure crystallized for me once I was running multiple support bots side by side for different apps. The first and third layers carry the app-specific differences; the middle layer ("polite, concise, no padding") becomes a shared template. When you revisit the prompt after months away, that separation makes the diffs readable.

import google.generativeai as genai
import os
 
genai.configure(api_key=os.environ["GEMINI_API_KEY"])
 
SYSTEM_PROMPT = """You are a friendly customer support agent for "TechShop."
 
[Role & Responsibilities]
- Answer product questions politely and accurately
- Explain troubleshooting steps in plain language
- Offer to escalate when you cannot solve the issue yourself
 
[Response Guidelines]
- Maintain a professional, never-pushy tone
- Present steps as short numbered items
- Do not speculate; admit when you don't know
- Avoid dumping multiple pieces of information at once
 
[Escalation Conditions]
- Complex refund or return cases
- Technical issues requiring deeper investigation
- The user explicitly requests a human
- Indications of complaint or legal concern
 
If you decide escalation is needed, append the keyword "[ESCALATE]" at the end of your reply."""

For more detail on prompt design, see the Gemini 2.5 Pro System Instructions Master Guide.

A Basic Multi-Turn Chatbot

start_chat() keeps conversation history for you. Two things to be aware of: a server restart loses the in-memory history, and a long conversation makes tokens balloon. I'll cover the token-bloat side in the cost section.

For persistence, a lightweight setup just serializes the history to Redis or Cloudflare KV keyed by session ID. I started in-memory and migrated to KV later; the rewrite wasn't pleasant, so designing for an external store from the start is the cheaper path.

import google.generativeai as genai
import os
 
genai.configure(api_key=os.environ["GEMINI_API_KEY"])
 
class CustomerSupportBot:
    def __init__(self, system_prompt: str):
        self.model = genai.GenerativeModel(
            model_name="gemini-2.5-flash",
            system_instruction=system_prompt,
        )
        self.chat = self.model.start_chat(history=[])
        self.escalated = False
        self.conversation_log = []
 
    def send_message(self, user_message: str) -> dict:
        if self.escalated:
            return {
                "response_text": "We're connecting you with a human agent. One moment please.",
                "escalate": True,
            }
        try:
            response = self.chat.send_message(user_message)
            response_text = response.text
            self.conversation_log.append({"role": "user", "content": user_message})
            self.conversation_log.append({"role": "assistant", "content": response_text})
            escalate = "[ESCALATE]" in response_text
            if escalate:
                response_text = response_text.replace("[ESCALATE]", "").strip()
                self.escalated = True
            return {"response_text": response_text, "escalate": escalate}
        except Exception:
            return {
                "response_text": "Sorry, we hit a temporary error. Please try again shortly.",
                "escalate": False,
            }

The key detail: once escalated is True, we stop calling the model and return a fixed message. Without that guard, the model occasionally takes back its own escalation decision on the next turn ("Actually, let me take care of this for you"), which destroys the user's trust because they no longer know who they're talking to.

Function Calling for FAQ Lookup

To suppress wrong facts, don't let the model answer from its own knowledge. Give it a FAQ database via Function Calling and let it pull authoritative answers when relevant. The basics are covered in Production Patterns for Function Calling with the Gemini 2.5 Pro API.

I keep the FAQ data in a Google Sheet that the operations team can edit themselves, then sync it to JSON every hour via Cloud Functions. The sync delay is mild compared with the benefit of letting non-engineers iterate on the answers.

faq_tool = {
    "function_declarations": [
        {
            "name": "search_faq",
            "description": "Search the FAQ database for product or service information.",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {"type": "string", "description": "Keyword or question to search"},
                },
                "required": ["query"],
            },
        }
    ]
}

The most common failure mode is the model ignoring an empty FAQ result and silently filling in from its own knowledge. Add one sentence to the system prompt: "If the tool returns nothing, do not guess—say you don't know and offer escalation." That single line drastically reduces hallucinated FAQ answers. In my own deployment I also force escalation whenever the tool returns empty, on the principle that a missing answer is itself a signal something needs human attention.

Escalation as the Last Line of Quality

If the bot cannot say "I don't know," users end up reading long, content-free paragraphs. Escalation isn't a cost-cutting feature—it's the last line of quality assurance.

def summarize_conversation(log):
    model = genai.GenerativeModel("gemini-2.5-flash")
    text = "\n".join(f"{m['role']}: {m['content']}" for m in log)
    prompt = f"Summarize the following conversation in 3 sentences or fewer. Include the issue, current state, and user mood.\n\n{text}"
    return model.generate_content(prompt).text
 
def assess_priority(summary):
    high = ["urgent", "refund", "lawsuit", "complaint", "fraud", "damaged"]
    return "high" if any(k in summary.lower() for k in high) else "normal"

In my own setup this fed a Slack notification: "high" priority pinged my phone immediately, everything else went to a morning digest. Average review time dropped to about three minutes per ticket and the after-hours phone buzz mostly stopped. The priority logic started as simple keyword matching and gradually absorbed phrasing variation over time—I never tried to make it perfect on day one.

Seven Pitfalls You Won't Find in the Official Docs

After more than half a year of production traffic, the following surprises became unmissable. I list them in roughly the order I hit them.

Here they are at a glance. If you arrived searching for one specific symptom, read only the paragraph that matches.

First, after a Function Calling result the model sometimes ignores the tool output and answers from its own knowledge. The fix is to add to the system prompt: "Tool results are the sole source of factual information; do not rely on anything else." Stability jumped immediately. The failure rate isn't high in isolation, but at scale it surfaces every single day, so build the guard in from the start.

Second, long conversations bloat tokens. Around 20 turns the latency becomes visibly worse. I added a "compression rotation" that summarizes the older portion of the log with Gemini Flash every 10 turns and replaces the front of history with the summary. Trigger by token count, not turn count, and latency stays roughly flat.

Third, the over-polite spiral I mentioned earlier. When users are angry, Gemini's tone-matching tendency makes it pile on "I completely understand…" preambles every turn, which paradoxically widens the distance. I now write into the prompt: "Use an apologetic preamble at most once per turn." It works better than you'd guess.

Fourth, an image attachment can break a text-only system prompt. Add a branch: "If an image is attached, summarize its contents in one line first, then answer the question." That keeps the response coherent.

Fifth, the post-escalation continuation problem. After emitting [ESCALATE], the model occasionally reverses its decision on the next turn. The implementation-level guard—stop calling the model once the flag is set—is the only reliable fix.

Sixth, mixing API keys between dev and prod. If you reuse a single key, your test calls inflate the production usage charts. Issue separate keys from Google AI Studio per environment; the dashboards immediately become trustworthy.

Seventh, empty-text responses. When the safety filter triggers, response.text can be empty, and you get a UI showing nothing where the bot's reply should be. Always check response.candidates[0].finish_reason and swap in a fixed fallback message when you see SAFETY or RECITATION. I lived through this one early, watching a customer's chat window show literally "..." while my stomach sank.

Putting History Compaction and the Empty-Response Guard in One Place

Pitfalls two and seven both live at the entrance and exit of the chat loop, so they can be absorbed by two small helpers. I keep these in a shared utility module and route every channel through them.

import google.generativeai as genai
 
SUMMARIZE_AFTER_TURNS = 10   # compact once history exceeds this
KEEP_RECENT_TURNS = 4        # leave the most recent turns verbatim
FALLBACK_TEXT = "Sorry — I wasn't able to answer that here. Let me connect you with a person."
 
 
def compact_history(history: list) -> list:
    """Fold older turns into a single summary, keeping recent turns intact."""
    if len(history) <= SUMMARIZE_AFTER_TURNS:
        return history
 
    old, recent = history[:-KEEP_RECENT_TURNS], history[-KEEP_RECENT_TURNS:]
    text = "\n".join(f"{m['role']}: {m['content']}" for m in old)
    model = genai.GenerativeModel("gemini-2.5-flash")
    summary = model.generate_content(
        "Summarize this support conversation as bullet points, keeping only "
        "unresolved issues and confirmed facts. Do not speculate or editorialize.\n\n" + text
    ).text
    return [{"role": "user", "content": f"[Conversation so far]\n{summary}"}] + recent
 
 
def safe_text(response) -> tuple[str, str | None]:
    """Never surface an empty reply. Returns (text, stop_reason)."""
    candidates = getattr(response, "candidates", None) or []
    if not candidates:
        return FALLBACK_TEXT, "NO_CANDIDATE"
 
    reason = getattr(candidates[0], "finish_reason", None)
    reason = getattr(reason, "name", str(reason))
    if reason in ("SAFETY", "RECITATION", "OTHER"):
        return FALLBACK_TEXT, reason
 
    text = (getattr(response, "text", "") or "").strip()
    if not text:
        return FALLBACK_TEXT, reason or "EMPTY"
    return text, None

The load-bearing detail in compact_history is the instruction not to speculate. Drop it and the summary starts carrying the model's interpretation, which the next turn then treats as established fact. I learned this when a summary confidently recorded "the user is requesting a refund" and nearly routed a plain feature question into the refund flow.

safe_text deliberately returns a tuple. The string shown in the UI and the reason written to logs serve different purposes. Count how often the fallback fires, broken down by reason, and a week where the safety filter starts misfiring becomes obvious immediately.

Cost Optimization and Operational Monitoring

For an indie-scale deployment, base everything on Flash and fall back to Pro only for hard cases. My setup generates a reply with Flash, inspects finish_reason and the response length, and re-generates with Pro only when something looks short, filtered, or otherwise off. Costs stay within the Flash budget and the hard-case accuracy approaches Pro.

At minimum, track three metrics: daily token spend, escalation rate, and average response token count. The cheapest pipeline is to ship Cloudflare Workers logs into BigQuery and graph it in Looker Studio. Glance at the dashboard each morning and you'll catch sudden spikes, or recurring weekday escalation patterns, far earlier than you would by reading logs alone.

For per-status error triage, see Classify Gemini API Errors by Status. For holding costs down with caching, see Gemini API Caching Strategy Notes.

Keeping Multilingual Support From Breaking

A common request is "answer in both Japanese and English." Gemini infers the language from context and switches, but leaving it entirely to inference produces surprises.

The most frequent one: a Japanese question with English proper nouns or error codes mixed in causes the whole reply to come back in English. I avoid this by writing into the response guidelines: "Match the primary language of the user's first message and keep using it across turns."

The other issue is machine-translated input from English-speaking users via Google Translate. The Japanese arrives with odd particles and missing subjects, and Gemini reads that as "a native speaker with idiosyncratic phrasing" and replies in equally awkward Japanese. I now run a pre-processing step that flags likely machine-translated input and switches the response back to the user's original language.

Testing the Bot's Response Quality

Long-running support bots need a steady way to test response quality. Before any release and at the weekly review I run a test set of 50–100 historical inquiries through the current system prompt and check that nothing regressed.

TEST_CASES = [
    {"input": "My order #12345 hasn't arrived yet", "expect_no_escalate": True, "expect_keywords": ["shipping"]},
    {"input": "I want a refund, I'm really upset", "expect_escalate": True},
]

Encoding expected keywords and escalation outcomes lets me re-run the suite with a single command after every prompt change. Even when the suite passes, I read through a sample by hand once a month—there are always subtle issues no test can catch.

Integrating With an Existing CRM or Contact Form

If you already have Zendesk or HubSpot, don't replace it—put the Gemini bot in front. The flow I prefer: the bot receives the inquiry first, uses Function Calling to read past transactions from the CRM, answers from the FAQ if possible, and creates a ticket in the CRM when escalation is needed. Critically, pass the full conversation history into the ticket body so the human agent doesn't have to re-read anything. That single decision dramatically lowers handoff friction.

For inquiries arriving via a contact form rather than real-time chat, asynchronous processing via Cloud Tasks or AWS SQS is more stable than synchronous Flash calls. A small added lead time is fine; peak-hour reliability improves and the overall user experience often gets better.

When the same bot serves multiple channels (email, web chat, in-app support), tag each entry point with a channel metadata field and write per-channel branches into the system prompt ("Email channel adds a one-line greeting"; "In-app support keeps replies short"). Fine-tuning the tone for each channel is best done weekly, from real logs.

A Phased Rollout That Actually Ships

Trying to build a perfect bot up front is the surest way to never launch. The three phases I run:

Phase one is "human approval mode": every bot reply is approved by a human before being sent. Two weeks of reading 200 or so logs gives you a tactile sense of what the bot gets wrong.

Phase two promotes the high-confidence categories (delivery status checks, canned FAQ) to fully automated, while keeping human review on the rest. For me, deliberately keeping "inquiries with image attachments" on the human side longer than I felt was necessary turned out to be exactly right.

Phase three slowly raises the automation ratio while watching the escalation rate and customer satisfaction (CSAT). If CSAT slips, identify which category dropped and rewrite that section of the system prompt. A 30-minute weekly review is enough to keep the loop turning.

Stick with it for six months and the support response time visibly shrinks. I've run app support solo alongside development for a long time, and ever since I added Gemini API to the loop, that heavy feeling of "I'm late on replies again" has receded.

Investing the time in the system frees you to actually develop, ship, and fix as one person—you stop being chased by support and start being the person who designs it. That shift, slow as it is, changes the landscape of indie development. I hope this notebook is useful for anyone wrestling with the same problem.