Building RAG Agents with Gemini × LlamaIndex — From Document Search to Multi-Step Reasoning

Configuring Gemini in LlamaIndex

Start by setting up the LLM and embedding model. The standard approach is to register them as global settings via the Settings object.

from llama_index.core import Settings
from llama_index.llms.gemini import Gemini
from llama_index.embeddings.gemini import GeminiEmbedding
 
# Configure the LLM (using Gemini 3.1 Pro)
Settings.llm = Gemini(
    model="models/gemini-3.1-pro",
    temperature=0.3,
    max_tokens=4096,
)
 
# Configure the embedding model
Settings.embed_model = GeminiEmbedding(
    model_name="models/text-embedding-004",
)
 
print("Gemini LLM and embedding model configured successfully")
# Output: Gemini LLM and embedding model configured successfully

The Gemini class wraps Google's google-genai SDK and supports Function Calling and streaming out of the box. Setting a lower temperature produces more factual, grounded responses — ideal for RAG use cases.

Loading Documents and Building the Index

Document Loading

LlamaIndex supports a wide range of data sources. Here's how to load local text files and PDFs:

from llama_index.core import SimpleDirectoryReader
 
# Load all documents from a directory
documents = SimpleDirectoryReader(
    input_dir="./docs",
    recursive=True,
    required_exts=[".txt", ".pdf", ".md"],
).load_data()
 
print(f"Loaded {len(documents)} documents")
# Example output: Loaded 42 documents

SimpleDirectoryReader auto-detects file formats and extracts text accordingly. For PDFs, it uses pypdf under the hood.

Building the Vector Index

With documents loaded, build a vector index. Each document is chunked, embedded using Gemini's embedding model, and indexed for similarity search.

from llama_index.core import VectorStoreIndex
 
# Build the index (chunk → embed → index)
index = VectorStoreIndex.from_documents(
    documents,
    show_progress=True,
)
 
print("Vector index built successfully")
# Output: Vector index built successfully

The defaults are 1024-token chunks with 200-token overlap. You can adjust these via Settings.chunk_size and Settings.chunk_overlap.

Running RAG Queries

With the index ready, create a query engine and start asking questions:

# Create a query engine
query_engine = index.as_query_engine(
    similarity_top_k=5,        # Retrieve top 5 similar chunks
    response_mode="compact",   # Generate concise responses
)
 
# Ask a question
response = query_engine.query(
    "What are the deployment steps for this project?"
)
 
print(response)
# Example output: The deployment steps are as follows. First...
 
# Inspect source documents
for node in response.source_nodes:
    print(f"  Source: {node.metadata.get('file_name', 'unknown')} "
          f"(score: {node.score:.3f})")
# Example output:
#   Source: deployment-guide.md (score: 0.892)
#   Source: ci-cd-pipeline.txt (score: 0.847)

The similarity_top_k parameter controls how many chunks are retrieved. Higher values improve coverage but increase context length and token costs. A range of 3–10 works well for most applications.

Measuring Chunk Size vs. Retrieval Accuracy — Comparing Three Settings

You'll often read that "the right chunk size depends on your use case" — but how much it actually depends is something you can only see by measuring. Using 42 of my own technical notes and how-to documents (roughly 180K tokens total), I changed only the chunk size, ran the same 20 questions, and recorded the share of queries where the correct chunk appeared in the top five (Recall@5), the average response time per query, and the number of embedding calls during the initial index build.

These numbers are one example from my local setup and will shift with the nature of your documents. Read them as a directional guide, not an absolute.

For my material, 512 tokens struck the best balance of accuracy and speed. At 256 the text was too fragmented and struggled with questions spanning multiple chunks; at 1024, noise crept into single chunks and unrelated fragments slipped into the top results. The point here isn't "512 is the answer" — it's the habit of running the same measurement once on your own documents. Changing the setting costs a few minutes; the confidence you gain pays off across the whole life of the system.

You can switch chunk settings with SentenceSplitter:

from llama_index.core.node_parser import SentenceSplitter
 
Settings.text_splitter = SentenceSplitter(
    chunk_size=512,
    chunk_overlap=64,
    paragraph_separator="\n\n",
)
 
# Always rebuild the index after changing settings
index = VectorStoreIndex.from_documents(documents, show_progress=True)

Building an Agent with Multi-Step Reasoning

Beyond simple RAG queries, you can build agents that combine multiple searches and reasoning steps. LlamaIndex's ReActAgent leverages Gemini's Function Calling to automatically select tools and reason step by step.

from llama_index.core.agent import ReActAgent
from llama_index.core.tools import QueryEngineTool, ToolMetadata
 
# Create tools from multiple indices
api_docs_tool = QueryEngineTool(
    query_engine=api_index.as_query_engine(similarity_top_k=3),
    metadata=ToolMetadata(
        name="api_docs",
        description="Searches the API reference documentation. "
                    "Can answer questions about endpoints, parameters, and response formats.",
    ),
)
 
tutorial_tool = QueryEngineTool(
    query_engine=tutorial_index.as_query_engine(similarity_top_k=3),
    metadata=ToolMetadata(
        name="tutorials",
        description="Searches tutorials and how-to guides. "
                    "Can answer questions about implementation steps and best practices.",
    ),
)
 
# Build the ReAct agent
agent = ReActAgent.from_tools(
    tools=[api_docs_tool, tutorial_tool],
    llm=Settings.llm,
    verbose=True,
    max_iterations=5,
)
 
# Ask the agent a complex question
response = agent.chat(
    "Check the user authentication API spec, then walk me through "
    "implementing a login feature using it."
)
print(response)
# Example output:
# Thought: I'll first check the API spec, then find the implementation tutorial
# Action: api_docs → Retrieve auth endpoint specifications
# Action: tutorials → Find login implementation guide
# Answer: The user authentication API uses POST /api/auth/login...

The ReActAgent cycles through "think → act → observe" iterations, automatically selecting the right tools to gather and synthesize information. The max_iterations parameter caps the number of reasoning steps to prevent infinite loops. The tool description is the agent's only cue for choosing a tool — when it's vague, the agent picks wrong, so the more concretely you name what each tool can find, the better the routing.

Implementing Hybrid Search (A Complete QueryFusionRetriever)

Vector search excels at semantic closeness, but it's weak on terms that can only be found through an exact match — part numbers, error codes, proper nouns. In my own setup, questions containing terms like TF-IDF or RFC 7231 sometimes missed the correct chunk when using vector search alone.

The fix is hybrid search, which combines vector similarity with keyword-based BM25 retrieval. Most write-ups mention it by name and stop there; here is the complete working code.

from llama_index.core.retrievers import QueryFusionRetriever
from llama_index.retrievers.bm25 import BM25Retriever
from llama_index.core.query_engine import RetrieverQueryEngine
 
# 1) Vector retriever
vector_retriever = index.as_retriever(similarity_top_k=5)
 
# 2) BM25 (keyword) retriever
bm25_retriever = BM25Retriever.from_defaults(
    docstore=index.docstore,
    similarity_top_k=5,
)
 
# 3) Fuse the two; combine scores via reciprocal rank fusion (RRF)
fusion_retriever = QueryFusionRetriever(
    retrievers=[vector_retriever, bm25_retriever],
    similarity_top_k=5,
    num_queries=1,             # 1 if you're not using query rewriting
    mode="reciprocal_rerank",  # combine via RRF
    use_async=True,
)
 
hybrid_engine = RetrieverQueryEngine.from_args(fusion_retriever)
 
response = hybrid_engine.query("How does RFC 7231 handle cache control?")
print(response)

mode="reciprocal_rerank" uses reciprocal rank fusion (RRF), which recomputes scores from the rank each retriever assigns. The benefit is that two result sets on different score scales get merged fairly, using rank as a shared yardstick. To use BM25Retriever, add pip install llama-index-retrievers-bm25.

Running the same 20 questions with hybrid search lifted Recall@5 from 0.86 to 0.91 on my material. Most of that gain came from questions with proper nouns or code, where BM25 caught what vector search missed. It isn't a cure-all — for tasks heavy on paraphrasing, it barely differs from vector search alone. If you adopt it, confirm on your own evaluation set which kinds of questions actually benefit before committing.

Reranking: The Move That Pays Off Once You Stop Losing Candidates

Once hybrid search widens the net, a different question surfaces: is the order of those top five results actually right? More than once I found the chunk BM25 had rescued sitting in fourth place — and that was the one I wanted the model to read.

The fix is straightforward. Retrieve generously, then reorder right before generation. In LlamaIndex, that reordering plugs in as a node postprocessor.

from llama_index.core.postprocessor import LLMRerank
from llama_index.core.query_engine import RetrieverQueryEngine
 
reranker = LLMRerank(
    choice_batch_size=5,
    top_n=5,          # how many survive the reordering and reach the model
)
 
# Retrieve 20, hand over only 5
rerank_engine = RetrieverQueryEngine.from_args(
    index.as_retriever(similarity_top_k=20),
    node_postprocessors=[reranker],
)
 
response = rerank_engine.query("How do I reissue a token after it expires?")
for node in response.source_nodes:
    print(f"{node.metadata.get('file_name')}: {node.score:.3f}")

Adding a reranker while leaving similarity_top_k at 5 buys you almost nothing — there are only five candidates to reorder. The benefit comes from the pairing: cast wide, deliver narrow. On my own corpus, retrieving 20 and narrowing to 5 clearly raised how often the correct chunk landed in first place.

The cost is that reranking triggers extra model calls proportional to the candidate count. Running the same 20 questions across three configurations, the tradeoff looked like this.

So reranking is a decision to buy accuracy with latency and spend. For internal search, where a few extra seconds are fine, it earns its keep. For a conversational UI that has to feel immediate, I found two workable compromises: keep similarity_top_k closer to 10 than 20, or move reranking behind a secondary "look deeper" action rather than running it on every turn.

One caveat: LLMRerank asks Gemini itself to do the ordering, so the judgment varies slightly between runs. When measuring against an evaluation set, I run each question a few times and look at the average rather than trusting a single pass.

Keeping Retrieval Quality on a Number — A Small Evaluation Loop

The piece most often neglected when moving RAG toward production is evaluation. If you keep improving on a vague "this feels better," you won't notice the day accuracy quietly slips. Before building any heavy evaluation harness, hand-write about 20 pairs of your own questions and their expected source, and run a small loop that measures recall.

# Prepare (question, filename that should contain the answer) pairs
eval_set = [
    ("What are the deployment steps?", "deployment-guide.md"),
    ("How long is the auth token valid?", "auth-spec.md"),
    # ... about 20 pairs
]
 
def recall_at_k(retriever, eval_set, k=5):
    hit = 0
    for question, expected_file in eval_set:
        nodes = retriever.retrieve(question)[:k]
        files = {n.metadata.get("file_name") for n in nodes}
        if expected_file in files:
            hit += 1
    return hit / len(eval_set)
 
score = recall_at_k(index.as_retriever(similarity_top_k=5), eval_set)
print(f"Recall@5 = {score:.2f}")
# Example output: Recall@5 = 0.86

Wire those 20 pairs into CI and you get an automatic guard against regressions whenever you change chunk settings or add documents. To assess the quality of the generated answer itself, LlamaIndex offers FaithfulnessEvaluator (is the answer faithful to the retrieved context?) and RelevancyEvaluator (does the answer address the question?), which can have Gemini itself act as judge. But LLM-based evaluation carries cost and variance, so it's safer to start by putting a number on retrieval-side recall first.

Persisting and Reloading Indices

Save your index to disk to avoid regenerating embeddings on every startup. This significantly reduces both startup time and API costs.

import os
 
PERSIST_DIR = "./storage"
 
# Save the index
index.storage_context.persist(persist_dir=PERSIST_DIR)
print(f"Index saved to {PERSIST_DIR}")
# Output: Index saved to ./storage
 
# Load the index (on subsequent runs)
from llama_index.core import StorageContext, load_index_from_storage
 
if os.path.exists(PERSIST_DIR):
    storage_context = StorageContext.from_defaults(persist_dir=PERSIST_DIR)
    index = load_index_from_storage(storage_context)
    print("Index loaded from disk")
    # Output: Index loaded from disk

For large document collections, embedding generation can incur significant API costs. If you're looking to optimize further, the caching strategies covered in Gemini API Context Caching Cost Optimization are worth exploring. To cut storage cost on the vector-DB side, the dimension reduction in Trimming Gemini Embedding Dimensions with a Matryoshka Approach is also worth considering.

Keeping the Index in Step When Documents Change

Persistence is where operation begins. Sooner or later a single runbook gets edited, or an outdated spec is swapped out. Rebuilding with VectorStoreIndex.from_documents every time you touch one file wastes both minutes and embedding budget.

The lever is giving each document a stable ID. With doc_id in place, LlamaIndex can work out what was added, what changed, and what disappeared.

from llama_index.core import SimpleDirectoryReader
 
# Pin doc_id to the file path so it stays identical across loads
documents = SimpleDirectoryReader(
    "./data",
    filename_as_id=True,
).load_data()
 
# Re-embed only what is new or modified
refreshed = index.refresh_ref_docs(documents)
print(f"Re-embedded: {sum(refreshed)} of {len(refreshed)} documents")
# Example output: Re-embedded: 3 of 128 documents
 
index.storage_context.persist(persist_dir=PERSIST_DIR)

refresh_ref_docs returns a boolean per document indicating whether it was new or had changed. If 3 out of 128 qualify, you pay for 3 embedding calls instead of 128 — and that gap widens with every document you add.

Deletions, however, are not handled for you. When a source file goes away, you have to remove it explicitly.

existing_ids = set(index.ref_doc_info.keys())
current_ids = {d.doc_id for d in documents}
 
for stale_id in existing_ids - current_ids:
    index.delete_ref_doc(stale_id, delete_from_docstore=True)
 
index.storage_context.persist(persist_dir=PERSIST_DIR)

Skip that step and a retired runbook keeps surfacing in results indefinitely. The answer sounds confident while quietly citing something that no longer applies — the hardest failure mode to notice — so I make sure refresh and deletion always finish inside the same job.

In a tool I run on my own, this sync happens as a nightly batch that logs how many documents were re-embedded and how many were dropped. Watching those counts turned out to be useful for a second reason. If a week that should have seen edits reports zero every night, something upstream has stopped feeding the pipeline. Before questioning retrieval quality, I now check whether the data ever arrived.

Operational Lessons the Official Docs Skip

The code above works, but running it in production surfaces practical pitfalls the tutorials never mention. Here are the ones I actually stumbled on, along with how I dealt with them.

First, wire in metadata filtering early. Being able to scope searches by department, date, or access level makes it far easier to add permission and freshness controls later. Some fields can't be attached without rebuilding the index, so it pays to include them in your initial schema.

from llama_index.core.vector_stores import MetadataFilters, ExactMatchFilter
 
filters = MetadataFilters(
    filters=[ExactMatchFilter(key="department", value="engineering")]
)
query_engine = index.as_query_engine(similarity_top_k=5, filters=filters)

Second, assume your embedding model is not a "set it once and forget it" choice. When the model updates, old and new vectors no longer live in the same space. Sending a new-model query against an index of old-dimension vectors makes retrieval quality drift downward, silently. I've written up this re-embedding design in When Your Firestore × Gemini Embeddings RAG Quietly Degrades.

Third, implement source attribution from day one. Simply attaching "which file and where this came from" to each answer buys you both early detection of hallucination and reader trust. Surfacing response.source_nodes in your UI takes only a few lines and pays off far beyond its cost.

Summary

Pairing the Gemini API with LlamaIndex lets you build an accurate RAG system grounded in your own documents, efficiently. Once the basic pipeline is running, the work is to keep going. Measure chunk size at least once. Pick up the misses with hybrid search. Push the near-misses up the ranking with a reranker when latency allows. Let incremental sync follow edits and deletions. Watch for drift with a small evaluation loop. That accumulation is, in my experience, what separates a prototype from a system you can run in production.

As a next step, I'd suggest preparing your own 20-question evaluation set. Holding a single numeric yardstick turns every later improvement from "intuition" into "verification." And if you want to push reasoning accuracy even further, the deep thinking techniques in Gemini 3 Deep Think Production Reasoning Patterns Guide are well worth exploring.

I'm still learning as I go, but I hope this helps with your own build. Thank you for reading.