One of the most common questions I get about building with Gemini is: "Which RAG approach should I use?"
Should you go with a straightforward vector search? Add query rewriting and reranking for better precision? Or build a fully agentic system where Gemini autonomously decides how many times to search? As the options multiply, so does the paralysis.
Here's what I've learned from working on multiple RAG projects: don't try to pick the "right" architecture upfront. Instead, understand what each pattern actually does—including the trade-offs—so you can make an informed decision based on what you observe in production. This article walks through all three patterns with working code, then gives you a practical framework for choosing.
Why Pattern Selection Gets Confusing
When you start reading RAG tutorials, you quickly hit a wall: "But what if the user's phrasing doesn't match the document language?" That concern leads you to Advanced RAG techniques, which introduce new complexity: "Is the added latency and cost worth it?"
The root of this confusion is trying to pick an architecture before you've measured anything. The pattern I follow on every project:
- Build Simple RAG and measure accuracy
- If accuracy falls short, identify whether the failure is in retrieval or generation
- Apply targeted improvements — Advanced RAG for retrieval issues, prompt tuning for generation issues
- Consider Agentic RAG only when multi-step reasoning is explicitly required
With that framing in mind, let's look at each pattern.
Pattern 1: Simple RAG — Running in 30 Minutes, But There's a Ceiling
The baseline: chunk your documents, embed them, embed the query, retrieve the top-k by cosine similarity, generate a response. Fast to build, easy to debug, and often good enough.
import google.generativeai as genai
import numpy as np
genai.configure(api_key="YOUR_GEMINI_API_KEY")
# Assumes chunks is a list of {"text": "...", "embedding": [...]}
# Pre-built by embedding your documents with task_type="RETRIEVAL_DOCUMENT"
def embed_text(text: str, task_type: str = "RETRIEVAL_DOCUMENT") -> list[float]:
result = genai.embed_content(
model="models/text-embedding-004",
content=text,
task_type=task_type,
)
return result["embedding"]
def cosine_similarity(a: list[float], b: list[float]) -> float:
a, b = np.array(a), np.array(b)
return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
def simple_rag(query: str, chunks: list[dict], top_k: int = 3) -> str:
# Use RETRIEVAL_QUERY for the user's question — asymmetric task type matters
query_embedding = embed_text(query, task_type="RETRIEVAL_QUERY")
scored = sorted(
chunks,
key=lambda c: cosine_similarity(query_embedding, c["embedding"]),
reverse=True,
)[:top_k]
context = "\n\n---\n\n".join(c["text"] for c in scored)
model = genai.GenerativeModel("gemini-2.5-flash")
prompt = f"""Answer the question using only the information provided below.
If the information doesn't contain the answer, say "I don't know."
Context:
{context}
Question: {query}"""
response = model.generate_content(prompt)
return response.textOne detail worth highlighting: using task_type="RETRIEVAL_QUERY" for the user's question and RETRIEVAL_DOCUMENT for your chunks is not just cosmetic. The text-embedding-004 model is trained for asymmetric retrieval, and getting this right measurably improves recall.
Where Simple RAG Falls Short
The most common failure mode is vocabulary mismatch. "How do I handle API errors?" and "What's the best retry strategy for rate limits?" are semantically close, but if the wording is different enough, cosine similarity might not surface the right chunk.
Chunking strategy is the other common issue. Splitting on character count without respecting sentence or section boundaries creates fragments that lose context, which hurts retrieval accuracy.
If either of these is causing your accuracy to drop below an acceptable threshold, that's the signal to look at Advanced RAG.
Pattern 2: Advanced RAG — Better Accuracy, Higher Implementation Cost
Advanced RAG improves on Simple RAG by enriching the query before searching and filtering the results after. Two techniques I've found consistently useful with Gemini: HyDE (Hypothetical Document Embeddings) for query expansion, and LLM-based reranking for result filtering.
HyDE works on an interesting insight: the embedding of a hypothetical answer to your question is often closer to the relevant document than the embedding of the question itself.
def generate_hypothetical_document(query: str) -> str:
"""Generate a hypothetical document that would answer the query (HyDE)."""
model = genai.GenerativeModel("gemini-2.5-flash")
prompt = f"""Write a passage of about 150 words that directly answers the following question.
It doesn't need to be factually accurate — this is used to improve search, not as an answer.
Question: {query}
Passage:"""
response = model.generate_content(prompt)
return response.text
def rerank_with_gemini(query: str, passages: list[str], top_n: int = 3) -> list[str]:
"""Use Gemini to select the most relevant passages."""
model = genai.GenerativeModel("gemini-2.5-flash")
numbered = "\n\n".join(f"[{i+1}] {p}" for i, p in enumerate(passages))
prompt = f"""Select the {top_n} passages most relevant to the question.
Return only the numbers, comma-separated, with no other text.
Question: {query}
Passages:
{numbered}
Most relevant {top_n} passage numbers:"""
response = model.generate_content(prompt)
indices_str = response.text.strip()
try:
indices = [int(x.strip()) - 1 for x in indices_str.split(",")]
return [passages[i] for i in indices if 0 <= i < len(passages)]
except Exception:
return passages[:top_n]
def advanced_rag_with_hyde(query: str, chunks: list[dict], top_k: int = 5) -> str:
# Step 1: Generate hypothetical document
hypothetical_doc = generate_hypothetical_document(query)
# Step 2: Embed the hypothetical document and retrieve
search_embedding = embed_text(hypothetical_doc, task_type="RETRIEVAL_QUERY")
scored = sorted(
chunks,
key=lambda c: cosine_similarity(search_embedding, c["embedding"]),
reverse=True,
)[:top_k]
# Step 3: Rerank to keep only the most relevant results
reranked = rerank_with_gemini(query, [c["text"] for c in scored], top_n=3)
context = "\n\n---\n\n".join(reranked)
# Step 4: Generate answer with a stronger model
model = genai.GenerativeModel("gemini-2.5-pro")
prompt = f"""Answer the question using only the information provided below.
Context:
{context}
Question: {query}"""
response = model.generate_content(prompt)
return response.textThe trade-off is visible in the code: each query now triggers at least two additional Gemini API calls (HyDE generation + reranking), before the final answer. That means roughly 3x the cost and latency of Simple RAG for the retrieval phase alone.
In my experience, Advanced RAG earns its keep in domains with specialized vocabulary (legal, medical, technical documentation) and in conversational interfaces where user input is short and ambiguous. For use cases where query and document vocabulary already align closely — product catalogs, internal wikis, FAQs — Simple RAG often delivers comparable accuracy at a fraction of the cost.
Pattern 3: Agentic RAG — Most Powerful, but Design Discipline Is Required
In Agentic RAG, the LLM autonomously decides when to search, what to search for, and how many times to search before producing an answer. Gemini's Function Calling makes this straightforward to implement.
import json
search_tool = {
"function_declarations": [
{
"name": "search_documents",
"description": "Search the document base and return relevant passages. Can be called multiple times with different queries.",
"parameters": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "The search query. More specific queries return better results.",
},
"top_k": {
"type": "integer",
"description": "Number of results to return (1-10).",
},
},
"required": ["query"],
},
}
]
}
def execute_search(query: str, chunks: list[dict], top_k: int = 3) -> str:
query_embedding = embed_text(query, task_type="RETRIEVAL_QUERY")
scored = sorted(
chunks,
key=lambda c: cosine_similarity(query_embedding, c["embedding"]),
reverse=True,
)[:top_k]
return "\n\n".join(c["text"] for c in scored)
def agentic_rag(question: str, chunks: list[dict], max_iterations: int = 5) -> str:
model = genai.GenerativeModel(
"gemini-2.5-pro",
tools=[search_tool], # type: ignore
)
messages = [{"role": "user", "parts": [question]}]
for iteration in range(max_iterations):
response = model.generate_content(messages)
candidate = response.candidates[0]
function_calls = [
p.function_call
for p in candidate.content.parts
if hasattr(p, "function_call") and p.function_call.name
]
if not function_calls:
# No tool calls — this is the final answer
return candidate.content.parts[0].text
# Append model turn to history
messages.append({"role": "model", "parts": candidate.content.parts})
# Execute tool calls and return results
tool_results = []
for fc in function_calls:
args = dict(fc.args)
result = execute_search(
query=args["query"],
chunks=chunks,
top_k=int(args.get("top_k", 3)),
)
tool_results.append(
{
"function_response": {
"name": fc.name,
"response": {"result": result},
}
}
)
messages.append({"role": "user", "parts": tool_results})
return "Failed to generate a response (iteration limit reached)"The max_iterations guard is non-negotiable. Without it, you risk runaway API usage. I'd also recommend logging the number of search calls per query in production — patterns where Gemini consistently needs 4+ iterations are a signal to refine your document structure or system prompt.
Where Agentic RAG genuinely shines: "Compare the error handling approaches in document A and document B," "What are the root causes of this issue, and what does the documentation say about each?" — questions that require gathering and synthesizing information from multiple searches.
For a deeper look at GraphRAG and knowledge graph patterns with Gemini, Gemini API × GraphRAG and Knowledge Graph Implementation covers the more advanced multi-hop reasoning architecture.
A Framework for Choosing
Here's how I'd map use cases to patterns:
Choose Simple RAG when:
- You're at prototype or MVP stage
- Query vocabulary closely matches document vocabulary (product catalogs, FAQs, internal wikis)
- Latency is critical (real-time chat)
- You want a performance baseline before deciding what to improve
Choose Advanced RAG when:
- User input is short, ambiguous, or conversational
- Your domain has specialized vocabulary that creates mismatch
- Simple RAG measurement shows accuracy below target
- The additional API cost per query (2-3x) is within budget
Choose Agentic RAG when:
- Questions require multi-step reasoning or synthesis across multiple searches
- Users have open-ended goals rather than specific factual queries
- Answer quality matters more than latency and cost predictability
For the complete production implementation — from vector store setup to deployment — Building a RAG System with Gemini API: Complete Implementation Guide covers the full stack. If you're looking at high-throughput pipelines, High-Speed RAG Pipeline with Gemini Flash is worth a read too.
Start by Measuring, Not by Architecting
My actual workflow on every new RAG project: build Simple RAG first and test it against 20-30 representative queries. Track the accuracy manually or with a simple rubric. If accuracy is above 80%, I ship it — there's no reason to add complexity that isn't justified by data.
If accuracy falls short, I look at where it fails. Retrieval failures (wrong chunks surfaced) point toward Advanced RAG techniques. Generation failures (right chunks, wrong answer) usually point toward prompt engineering or model selection.
Agentic RAG comes into the picture only after I've identified questions that genuinely require multi-hop reasoning and confirmed that simpler approaches can't handle them.
The fastest path to a working RAG system is almost always: build the simplest thing, measure it honestly, and improve specifically where it falls short.