Building RAG Agents with Gemini × LlamaIndex — From Document Search to Multi-Step Reasoning

Configuring Gemini in LlamaIndex

Start by setting up the LLM and embedding model. The standard approach is to register them as global settings via the Settings object.

from llama_index.core import Settings
from llama_index.llms.gemini import Gemini
from llama_index.embeddings.gemini import GeminiEmbedding
 
# Configure the LLM (using Gemini 3.1 Pro)
Settings.llm = Gemini(
    model="models/gemini-3.1-pro",
    temperature=0.3,
    max_tokens=4096,
)
 
# Configure the embedding model
Settings.embed_model = GeminiEmbedding(
    model_name="models/text-embedding-004",
)
 
print("Gemini LLM and embedding model configured successfully")
# Output: Gemini LLM and embedding model configured successfully

The Gemini class wraps Google's google-genai SDK and supports Function Calling and streaming out of the box. Setting a lower temperature produces more factual, grounded responses — ideal for RAG use cases.

Loading Documents and Building the Index

Document Loading

LlamaIndex supports a wide range of data sources. Here's how to load local text files and PDFs:

from llama_index.core import SimpleDirectoryReader
 
# Load all documents from a directory
documents = SimpleDirectoryReader(
    input_dir="./docs",
    recursive=True,
    required_exts=[".txt", ".pdf", ".md"],
).load_data()
 
print(f"Loaded {len(documents)} documents")
# Example output: Loaded 42 documents

SimpleDirectoryReader auto-detects file formats and extracts text accordingly. For PDFs, it uses pypdf under the hood.

Building the Vector Index

With documents loaded, build a vector index. Each document is chunked, embedded using Gemini's embedding model, and indexed for similarity search.

from llama_index.core import VectorStoreIndex
 
# Build the index (chunk → embed → index)
index = VectorStoreIndex.from_documents(
    documents,
    show_progress=True,
)
 
print("Vector index built successfully")
# Output: Vector index built successfully

The defaults are 1024-token chunks with 200-token overlap. You can adjust these via Settings.chunk_size and Settings.chunk_overlap.

Running RAG Queries

With the index ready, create a query engine and start asking questions:

# Create a query engine
query_engine = index.as_query_engine(
    similarity_top_k=5,        # Retrieve top 5 similar chunks
    response_mode="compact",   # Generate concise responses
)
 
# Ask a question
response = query_engine.query(
    "What are the deployment steps for this project?"
)
 
print(response)
# Example output: The deployment steps are as follows. First...
 
# Inspect source documents
for node in response.source_nodes:
    print(f"  Source: {node.metadata.get('file_name', 'unknown')} "
          f"(score: {node.score:.3f})")
# Example output:
#   Source: deployment-guide.md (score: 0.892)
#   Source: ci-cd-pipeline.txt (score: 0.847)

The similarity_top_k parameter controls how many chunks are retrieved. Higher values improve coverage but increase context length and token costs. A range of 3–10 works well for most applications.

Measuring Chunk Size vs. Retrieval Accuracy — Comparing Three Settings

You'll often read that "the right chunk size depends on your use case" — but how much it actually depends is something you can only see by measuring. Using 42 of my own technical notes and how-to documents (roughly 180K tokens total), I changed only the chunk size, ran the same 20 questions, and recorded the share of queries where the correct chunk appeared in the top five (Recall@5), the average response time per query, and the number of embedding calls during the initial index build.

These numbers are one example from my local setup and will shift with the nature of your documents. Read them as a directional guide, not an absolute.

For my material, 512 tokens struck the best balance of accuracy and speed. At 256 the text was too fragmented and struggled with questions spanning multiple chunks; at 1024, noise crept into single chunks and unrelated fragments slipped into the top results. The point here isn't "512 is the answer" — it's the habit of running the same measurement once on your own documents. Changing the setting costs a few minutes; the confidence you gain pays off across the whole life of the system.

You can switch chunk settings with SentenceSplitter:

from llama_index.core.node_parser import SentenceSplitter
 
Settings.text_splitter = SentenceSplitter(
    chunk_size=512,
    chunk_overlap=64,
    paragraph_separator="\n\n",
)
 
# Always rebuild the index after changing settings
index = VectorStoreIndex.from_documents(documents, show_progress=True)

Building an Agent with Multi-Step Reasoning

Beyond simple RAG queries, you can build agents that combine multiple searches and reasoning steps. LlamaIndex's ReActAgent leverages Gemini's Function Calling to automatically select tools and reason step by step.

from llama_index.core.agent import ReActAgent
from llama_index.core.tools import QueryEngineTool, ToolMetadata
 
# Create tools from multiple indices
api_docs_tool = QueryEngineTool(
    query_engine=api_index.as_query_engine(similarity_top_k=3),
    metadata=ToolMetadata(
        name="api_docs",
        description="Searches the API reference documentation. "
                    "Can answer questions about endpoints, parameters, and response formats.",
    ),
)
 
tutorial_tool = QueryEngineTool(
    query_engine=tutorial_index.as_query_engine(similarity_top_k=3),
    metadata=ToolMetadata(
        name="tutorials",
        description="Searches tutorials and how-to guides. "
                    "Can answer questions about implementation steps and best practices.",
    ),
)
 
# Build the ReAct agent
agent = ReActAgent.from_tools(
    tools=[api_docs_tool, tutorial_tool],
    llm=Settings.llm,
    verbose=True,
    max_iterations=5,
)
 
# Ask the agent a complex question
response = agent.chat(
    "Check the user authentication API spec, then walk me through "
    "implementing a login feature using it."
)
print(response)
# Example output:
# Thought: I'll first check the API spec, then find the implementation tutorial
# Action: api_docs → Retrieve auth endpoint specifications
# Action: tutorials → Find login implementation guide
# Answer: The user authentication API uses POST /api/auth/login...

The ReActAgent cycles through "think → act → observe" iterations, automatically selecting the right tools to gather and synthesize information. The max_iterations parameter caps the number of reasoning steps to prevent infinite loops. The tool description is the agent's only cue for choosing a tool — when it's vague, the agent picks wrong, so the more concretely you name what each tool can find, the better the routing.

Implementing Hybrid Search (A Complete QueryFusionRetriever)

Vector search excels at semantic closeness, but it's weak on terms that can only be found through an exact match — part numbers, error codes, proper nouns. In my own setup, questions containing terms like TF-IDF or RFC 7231 sometimes missed the correct chunk when using vector search alone.

The fix is hybrid search, which combines vector similarity with keyword-based BM25 retrieval. Most write-ups mention it by name and stop there; here is the complete working code.

from llama_index.core.retrievers import QueryFusionRetriever
from llama_index.retrievers.bm25 import BM25Retriever
from llama_index.core.query_engine import RetrieverQueryEngine
 
# 1) Vector retriever
vector_retriever = index.as_retriever(similarity_top_k=5)
 
# 2) BM25 (keyword) retriever
bm25_retriever = BM25Retriever.from_defaults(
    docstore=index.docstore,
    similarity_top_k=5,
)
 
# 3) Fuse the two; combine scores via reciprocal rank fusion (RRF)
fusion_retriever = QueryFusionRetriever(
    retrievers=[vector_retriever, bm25_retriever],
    similarity_top_k=5,
    num_queries=1,             # 1 if you're not using query rewriting
    mode="reciprocal_rerank",  # combine via RRF
    use_async=True,
)
 
hybrid_engine = RetrieverQueryEngine.from_args(fusion_retriever)
 
response = hybrid_engine.query("How does RFC 7231 handle cache control?")
print(response)

mode="reciprocal_rerank" uses reciprocal rank fusion (RRF), which recomputes scores from the rank each retriever assigns. The benefit is that two result sets on different score scales get merged fairly, using rank as a shared yardstick. To use BM25Retriever, add pip install llama-index-retrievers-bm25.

Running the same 20 questions with hybrid search lifted Recall@5 from 0.86 to 0.91 on my material. Most of that gain came from questions with proper nouns or code, where BM25 caught what vector search missed. It isn't a cure-all — for tasks heavy on paraphrasing, it barely differs from vector search alone. If you adopt it, confirm on your own evaluation set which kinds of questions actually benefit before committing.

Keeping Retrieval Quality on a Number — A Small Evaluation Loop

The piece most often neglected when moving RAG toward production is evaluation. If you keep improving on a vague "this feels better," you won't notice the day accuracy quietly slips. Before building any heavy evaluation harness, hand-write about 20 pairs of your own questions and their expected source, and run a small loop that measures recall.

# Prepare (question, filename that should contain the answer) pairs
eval_set = [
    ("What are the deployment steps?", "deployment-guide.md"),
    ("How long is the auth token valid?", "auth-spec.md"),
    # ... about 20 pairs
]
 
def recall_at_k(retriever, eval_set, k=5):
    hit = 0
    for question, expected_file in eval_set:
        nodes = retriever.retrieve(question)[:k]
        files = {n.metadata.get("file_name") for n in nodes}
        if expected_file in files:
            hit += 1
    return hit / len(eval_set)
 
score = recall_at_k(index.as_retriever(similarity_top_k=5), eval_set)
print(f"Recall@5 = {score:.2f}")
# Example output: Recall@5 = 0.86

Wire those 20 pairs into CI and you get an automatic guard against regressions whenever you change chunk settings or add documents. To assess the quality of the generated answer itself, LlamaIndex offers FaithfulnessEvaluator (is the answer faithful to the retrieved context?) and RelevancyEvaluator (does the answer address the question?), which can have Gemini itself act as judge. But LLM-based evaluation carries cost and variance, so it's safer to start by putting a number on retrieval-side recall first.

Persisting and Reloading Indices

Save your index to disk to avoid regenerating embeddings on every startup. This significantly reduces both startup time and API costs.

import os
 
PERSIST_DIR = "./storage"
 
# Save the index
index.storage_context.persist(persist_dir=PERSIST_DIR)
print(f"Index saved to {PERSIST_DIR}")
# Output: Index saved to ./storage
 
# Load the index (on subsequent runs)
from llama_index.core import StorageContext, load_index_from_storage
 
if os.path.exists(PERSIST_DIR):
    storage_context = StorageContext.from_defaults(persist_dir=PERSIST_DIR)
    index = load_index_from_storage(storage_context)
    print("Index loaded from disk")
    # Output: Index loaded from disk

For large document collections, embedding generation can incur significant API costs. If you're looking to optimize further, the caching strategies covered in Gemini API Context Caching Cost Optimization are worth exploring. To cut storage cost on the vector-DB side, the dimension reduction in Trimming Gemini Embedding Dimensions with a Matryoshka Approach is also worth considering.

Operational Lessons the Official Docs Skip

The code above works, but running it in production surfaces practical pitfalls the tutorials never mention. Here are the ones I actually stumbled on, along with how I dealt with them.

First, wire in metadata filtering early. Being able to scope searches by department, date, or access level makes it far easier to add permission and freshness controls later. Some fields can't be attached without rebuilding the index, so it pays to include them in your initial schema.

from llama_index.core.vector_stores import MetadataFilters, ExactMatchFilter
 
filters = MetadataFilters(
    filters=[ExactMatchFilter(key="department", value="engineering")]
)
query_engine = index.as_query_engine(similarity_top_k=5, filters=filters)

Second, assume your embedding model is not a "set it once and forget it" choice. When the model updates, old and new vectors no longer live in the same space. Sending a new-model query against an index of old-dimension vectors makes retrieval quality drift downward, silently. I've written up this re-embedding design in When Your Firestore × Gemini Embeddings RAG Quietly Degrades.

Third, implement source attribution from day one. Simply attaching "which file and where this came from" to each answer buys you both early detection of hallucination and reader trust. Surfacing response.source_nodes in your UI takes only a few lines and pays off far beyond its cost.

Summary

Pairing the Gemini API with LlamaIndex lets you build an accurate RAG system grounded in your own documents, efficiently. Once the basic pipeline is running, don't stop there: measure chunk size at least once, pick up the misses with hybrid search where needed, and guard against drift with a small evaluation loop. These three steps of going deeper are, in my experience, what separates a prototype from a system you can run in production.

As a next step, I'd suggest preparing your own 20-question evaluation set. Holding a single numeric yardstick turns every later improvement from "intuition" into "verification." And if you want to push reasoning accuracy even further, the deep thinking techniques in Gemini 3 Deep Think Production Reasoning Patterns Guide are well worth exploring.

I'm still learning as I go, but I hope this helps with your own build. Thank you for reading.