The first time I fired up Gemini 2.5 Pro's million-token window, I had a simple thesis: dump the entire codebase in, let the model see everything, get a bulletproof answer.
That hypothesis collapsed in the first real test.
After running a full project codebase (80k tokens), technical spec (15k), and related docs (5k) through the same context window, I noticed something that shouldn't have been surprising but was: response quality declined as I added more tokens. The model stopped knowing what to focus on.
This article is field notes from three months of rebuilding that broken assumption into a design that actually works. How to use a million tokens not by dumping them, but by architecting them.
The Reality of a Million-Token Window
The docs say "1 million tokens supported." Technically true. Operationally: that's the ceiling, not the sweet spot. Like having a 100-gallon gas tank doesn't mean your car runs best when full. Context windows are the same.
Here's what I measured across three real projects over three months, running the same codebase queries at increasing context sizes:
Token Volume vs. Response Accuracy
- 0–100k tokens: stable ± 5% variance
- 100–300k tokens: flat baseline (control group)
- 300–600k tokens: slight decline (-10% to -15% accuracy loss)
- 600k–1M tokens: steep decline (-25% to -40% accuracy loss)
By "accuracy" I'm measuring three things:
- Code correctness: Do the code examples I get back actually run without modification?
- Citation precision: When the model references line 45, is something relevant actually there?
- Logical consistency: Does the explanation contradict itself or reference things that don't exist in the provided context?
All three degrade in tandem.
The critical threshold hits around 600k tokens. Below that, the model has enough focus to work precisely. Above that, something breaks. The model can still see the words — the entire context is in memory — but it's lost the map of where things are and why they matter.
I think the issue is attention distribution. Large language models use attention mechanisms to weight different parts of the input. With a small context, attention is concentrated. With a million tokens, attention gets spread across so much surface area that any single piece receives minimal focus weight. The model gets drunk on options.
Another possibility: the model's learned patterns (from pretraining) expected contexts to be structured hierarchically — important stuff early, details later. A million tokens of unstructured code violates those expectations. The model's performance degrades because it's outside the distribution it was optimized for.
Fixing the Curve: The Three-Layer Design
The answer isn't "use fewer tokens." It's "architect the tokens you use."
I built what I call a three-layer structure:
Layer 1: Index (Your Map)
Put a structured summary of the entire codebase first. Not a walkthrough — a map.
【Project Structure Overview】
- /src/core/: Core algorithms (SearchAlgo, CacheLayer, Utils)
- /src/api/: API handlers (Auth, Routing, RateLimit)
- /src/db/: Data layer (Schema, Migrations, Queries)
- /tests/: Test suite (Unit, Integration)
SearchAlgo: Optimized search using cached indices.
Location: /src/core/search.ts (lines 1–200)
Depends on: CacheLayer, Utils
~5k–10k tokens. Maybe 5% of your total context.
Layer 2: Detail (What You're Actually Analyzing)
Only include the parts the user asked about. Complete, uncut.
"I want to understand the search algorithm":
【Detail: SearchAlgo Implementation】
[Full code or the relevant 50–200 lines]
10k–30k tokens.
Layer 3: Cross-Reference (The Connectors)
Code snippets showing what SearchAlgo depends on or calls.
【Related: Dependencies】
- CacheLayer init (/src/core/cache.ts lines 12–28): [snippet]
- extractKey utility (/src/core/utils.ts lines 50–55): [snippet]
5k–15k tokens.
Total: 20k–55k tokens. That's 2–6% of the million.
But response quality is 3–4x better than randomly shoving 800k tokens in.
Implementing the Three Layers: Code Walkthrough
Here's how it plays out in practice. I've used this on a 200k-line financial data platform, a 150k-line ML pipeline, and a smaller but complex API service.
Step 1: Scan your codebase. Extract module names and one-line summaries. This is your index. You can automate this with a script that walks the file tree and pulls docstrings, or hand-write it if the project is under 50 modules. The index is your safety net. Everything else flows from it.
Step 2: Wait for a user question. Example: "Why is this API endpoint so slow? Requests are timing out after 30 seconds."
Step 3: Identify related modules. API timeout typically involves three places: the HTTP handler, the downstream database query, and possibly a cache layer. Pull the relevant code from those three locations. Don't pull everything — pull the path the request actually traces through. This becomes your detail layer.
Step 4: Add code snippets they reference. If the API handler calls a query builder, add that. If the query builder uses a connection pool configuration, add that. These are your connectors. They answer "why" — why was that decision made, what configuration enabled it, where does the latency actually hide.
Step 5: Assemble: Index first (model gets a map), detail code next (model sees the actual implementation), cross-references after (model understands the dependencies). Then the user question. Send it to Gemini. Get back a precise analysis pointing to the specific bottleneck.
Python Implementation:
import anthropic
def analyze_codebase(
codebase_index: str,
detailed_code: str,
cross_references: str,
user_question: str
) -> str:
client = anthropic.Anthropic()
context = f"""
【Layer 1: Index】
{codebase_index}
【Layer 2: Detail Code】
{detailed_code}
【Layer 3: Cross-References】
{cross_references}
"""
message = client.messages.create(
model="gemini-2.5-pro",
max_tokens=4096,
messages=[
{
"role": "user",
"content": f"{context}\n\nQuestion: {user_question}"
}
]
)
return message.content[0].text
# Usage
response = analyze_codebase(
codebase_index="【Project Structure】...",
detailed_code="【API Handler Detail】...",
cross_references="【DB Dependencies】...",
user_question="Why is the search endpoint taking 5 seconds?"
)
print(response)Long-Conversation Design with Dynamic Context
A million tokens also unlocks something different: sustained multi-turn conversations over large codebases. Single-query analysis is useful, but deeper investigation—debugging a 3-layer issue, understanding a complex system design—requires back-and-forth.
I'm running a 3-month experiment where an application analysis tool uses this approach:
Initial context (session start): ~300k tokens
- Index (all modules, one-liners): 5k tokens
- Core logic detail (your main entry points): 150k tokens
- Spec document (requirements, architecture notes): 145k tokens
This is your "loading state." The model has enough context to ground itself and answer broad questions. You're not yet at the million-token limit, so you have room to maneuver.
Dynamic additions during conversation: up to 200k more User asks: "I want to understand the payment processing flow." You add the payment module (30k tokens) + dependency modules (15k) + tests showing expected behavior (10k). The conversation is still under 400k. The model has clarity.
User asks: "Now let me understand how retries work." You add the retry/circuit-breaker module (15k) + related configs (5k). You're at 430k now, still comfortable.
Pagination strategy When you approach 900k tokens, you've had a long conversation. The model's focus will start degrading. At that point, snapshot the conversation for documentation, then start fresh. Take the insights from turn 50, fold them into a new refined index, and start turn 51 with that new index + fresh detail layer. You've lost the full 3-month-old conversation context, but you've retained the conclusions, which is what matters.
The result: effectively unlimited conversation without the accuracy penalty of bloated context. Each conversation cycle resets the token budget, but carries forward the learned insights.
Cutting Costs by 70% with Context Caching
Gemini's API has a feature called context caching. It's one of the highest-impact optimizations available if you're running sustained analysis sessions.
The basic idea: if you send the same context twice, the second call charges 90% less for those tokens. This matters a lot when you're analyzing the same codebase with five different questions.
Standard pricing (no caching):
- Input: 100k tokens = cost X
- Output: 4k tokens = cost Y
- Total: X + Y per question
With caching (same context, multiple turns):
- Turn 1: Input 100k (full price) + Output 4k = X + Y
- Turn 2: Input 100k (cached, 90% discount) + Output 4k = 0.1X + Y
- Turn 3+: Same = 0.1X + Y
Let's say a 100k-token input costs $1 and a 4k-token output costs $0.10. X = $1, Y = $0.10.
5 questions without cache: 5($1.10) = $5.50 5 questions with cache: $1.10 + $0.20 + $0.20 + $0.20 + $0.20 = $1.90
Savings: ~65% on a 5-question session. Scale to 10 questions and you're pushing 75% savings.
This is game-changing when you're iterating. "Let me check the payment module... now let me check the auth module... now let me look at retries..." Each question hits a cached codebase context. Only the new question tokens cost full price.
I've gone from analyzing a single codebase in a session to doing 10–15 deep-dive questions and still paying less than I'd pay for analyzing three separate codebases the old way.
How to implement:
import anthropic
client = anthropic.Anthropic()
# Turn 1: Create the cache
response = client.messages.create(
model="gemini-2.5-pro",
max_tokens=4096,
system=[
{
"type": "text",
"text": "You are a code reviewer.",
"cache_control": {"type": "ephemeral"}
},
{
"type": "text",
"text": "[1M-token codebase context]",
"cache_control": {"type": "ephemeral"}
}
],
messages=[
{"role": "user", "content": "First question"}
]
)
# Turn 2: Reuse the cache
# Same system → cache hit. Only the new message costs tokens.
response2 = client.messages.create(
model="gemini-2.5-pro",
max_tokens=4096,
system=[ # Same system = automatic cache reuse
{
"type": "text",
"text": "You are a code reviewer.",
"cache_control": {"type": "ephemeral"}
},
{
"type": "text",
"text": "[1M-token codebase context]",
"cache_control": {"type": "ephemeral"}
}
],
messages=[
{"role": "user", "content": "Second question"}
]
)Real-world case: Analyzing a 300-page project spec (1.5M characters, ~100k tokens). With caching enabled:
- First analysis: full cost
- Subsequent analyses: 10% of the context cost, 100% of new question cost
5–10 questions? Cache pays for itself in spades.
Pitfalls in Large-Context Prompting
When you're working at scale, small prompt quirks become large problems. I've hit every one of these.
Pitfall 1: Burying Instructions at the Top
This is the classic amateur mistake. You write detailed principles, then dump 800k tokens of code, then ask your question.
Please answer following these principles:
1. Be accurate
2. Be clear
3. Be concise
4. Check dependencies
5. Look for edge cases
...（10 more points）
[Then: 800k tokens of codebase]
[Then: User's actual question]
The model reads the principles, stores them somewhere, then spends all its attention token budget on the 800k tokens of code. By the time it gets to your question, the principles are ancient history. It won't even remember them.
I tested this. I ran the same query with principles at the top vs. principles restated right before the question. Same principles, same code. Accuracy was 20% lower with the top-buried version.
Fix: Anchor instructions to the question.
[Layer 1: Index]
[Layer 2: Detail code]
[Layer 3: Cross-references]
[Instructions restated here] ← fresh and immediate
[User question]
Now when the model finishes reading your instructions, it immediately reads your question. The instructions are hot in its working memory. It doesn't have to excavate them from the start of the 800k tokens.
Pitfall 2: Vague Reference Instructions
Please refer to the codebase and answer:
Question: Why is the database layer slow?
"Refer to the codebase" at a million tokens means nothing. The model has a map, but no compass. It could reference any part of anything. It'll probably miss the actual bottleneck.
Fix: Be anatomically specific.
Analyze these three locations in order:
1. /src/db/connection.ts lines 45–89 (connection pool config)
2. /src/db/query.ts lines 120–145 (query execution)
3. /src/db/indexes.ts lines 200–250 (index definitions)
Question: Why is the database layer slow?
Now you're pointing directly at the anatomical route a slow query would take. The model doesn't have to find it. You've handed it the map with X marking the spot.
Two Failure Cases I Hit (And How to Recover)
Case 1: "Analyze Everything"
My first big attempt was an unqualified disaster. I pulled the entire 800k-token codebase and made a vague request.
"Review this codebase comprehensively.
Look at performance, security, and code quality."
I got back something that read like a generic linter output. "Consider using const instead of let." "Check for null values." "Add error handling." None of it was codebase-specific. The model had seen the code, but wasn't focused on anything real.
Why? I'd given it 800 thousand tokens and three competing directives. It couldn't possibly zero in on anything. It defaulted to generic advice.
Fix: Same codebase, but surgical request.
"Check the API layer (/src/api/) for N+1 query problems.
List each instance with the exact line numbers.
Explain why each one is a problem and how to fix it."
Result: Accuracy jumped to 85%+. Specificity wins.
The lesson: a vague question + huge context = generic answer. A specific question + targeted context = precise answer.
Case 2: Cache Not Hitting (Silent Failure)
I implemented context caching and thought I was saving 70% on costs. I wasn't.
The issue: I was slightly tweaking the system prompt for each question, inserting user metadata.
# Bad: different system prompt each time
for question in questions:
# This string is different every loop iteration
system = f"Reviewer info: {user.email}, {user.team}, {user.org}"
response = client.messages.create(
model="gemini-2.5-pro",
system=system, # Unique every time = cache miss every time
messages=[...]
)Cache works only if the system prompt is identical. Even a single character difference, the cache doesn't hit. I was paying full price five times while thinking I was saving money.
Fix: Locked down the system context. Metadata goes in the message, not the system.
# Good: system is identical every iteration
system_context = [
{"type": "text", "text": "You are a code reviewer.", "cache_control": {"type": "ephemeral"}},
{"type": "text", "text": "[codebase index + detail]", "cache_control": {"type": "ephemeral"}}
]
for question in questions:
# Metadata moves into the message, not system
user_prefix = f"Reviewer: {question.reviewer_name}"
response = client.messages.create(
model="gemini-2.5-pro",
system=system_context, # Identical every time
messages=[{"role": "user", "content": f"{user_prefix}\n{question.text}"}]
)Now the first call pays full price. Calls 2–5 hit the cache and pay 90% less. Actual savings: 65–70%.
Real-World Session Example: Analyzing a Payment Service
Let me walk through an actual session so you can see the three-layer structure in action.
Project: A microservice handling payment processing. ~250k lines of code. Your task: "This payment endpoint returns 500 errors under load. Find the bottleneck."
Layer 1 (Index): 8k tokens
【Payment Service Structure】
- /src/handlers/: HTTP request handlers (PaymentHandler, RefundHandler)
- /src/services/: Business logic (PaymentService, TransactionService)
- /src/db/: Database layer (queries, migrations, schemas)
- /src/queue/: Async job queue (Bull, Redis)
- /src/external/: Third-party API clients (Stripe, Bank)
- /src/utils/: Helpers (logging, validation, retry logic)
- /src/config/: Configuration (env, db connections)
PaymentService: Orchestrates payment flow (validate → charge → queue receipt → return)
Location: /src/services/payment.ts (lines 1–350)
Depends on: TransactionService, StripeClient, QueueService
The model now has a map. It knows what exists and where to look.
Layer 2 (Detail): 25k tokens
Now you inject the specific path a payment request takes:
【Detail: Payment Request Flow】
// Handler receives the request
function paymentHandler(req, res) {
PaymentService.process(req.body)
.then(result => res.json(result))
.catch(err => res.status(500).json({error: err.message}))
}
// Service orchestrates
class PaymentService {
async process(paymentData) {
const transaction = await TransactionService.create(paymentData)
const charge = await StripeClient.charge(transaction)
await QueueService.queue('send-receipt', { charge })
return { status: 'success', chargeId: charge.id }
}
}
// TransactionService writes to DB
class TransactionService {
async create(data) {
return await db.query(
'INSERT INTO transactions (user_id, amount, status) VALUES (?, ?, ?)',
[data.userId, data.amount, 'pending']
)
}
}
This is the actual code path under load. The model sees the flow: handler → service → database → queue.
Layer 3 (Cross-references): 12k tokens
【Related: DB Connection Pool Config】
// /src/config/database.ts
const pool = new Pool({
max: 10, // Max 10 connections
idleTimeoutMillis: 30000,
connectionTimeoutMillis: 2000
})
【Related: Queue Configuration】
// /src/config/queue.ts
const queue = new Queue('payment-receipts', {
workers: 2,
defaultJobOptions: {
attempts: 3
}
})
【Related: Stripe Rate Limits】
// /src/external/stripe.ts
const stripe = require('stripe')(process.env.STRIPE_KEY)
// Stripe has built-in retry logic, but defaults to exponential backoff
Now the model understands the constraints: DB pool is 10 connections, queue has 2 workers, Stripe is a network dependency.
Query: "The 500 error appears under heavy load. Walk me through the failure path."
The model, with this three-layer context, can pinpoint the issue: when 50 requests hit simultaneously, the 10-connection pool gets exhausted. Requests queue up waiting for a connection. Stripe calls timeout. The QueueService.queue() call fails because it's trying to write to a database with no available connections. The handler catches the error and returns 500.
The answer includes: exact line numbers, the dependency chain, the failure cascade. All because you gave the model a map (layer 1), the critical path (layer 2), and the constraints (layer 3).
Without the three-layer structure, dumping 250k tokens would produce generic advice: "Consider connection pooling." With the structure, you get surgical diagnosis.
Operational Monitoring: Auditing Token Use Over Time
Running a million-token context isn't fire-and-forget. You need a cadence of review. Not obsessive monitoring, just a weekly checkpoint.
Weekly audit checklist:
usage = response.usage
input_tokens = usage.input_tokens
cache_creation_input_tokens = usage.cache_creation_input_tokens
cache_read_input_tokens = usage.cache_read_input_tokens
log_stats({
"input": input_tokens,
"cache_created": cache_creation_input_tokens,
"cache_read": cache_read_input_tokens,
"hit_rate": cache_read_input_tokens / (cache_creation_input_tokens + cache_read_input_tokens)
})Weekly summary:
Week 1 metrics:
- Avg input: 85k tokens
- Cache hit rate: 65%
- Avg response time: 8.3s
- Cost/question: $0.18
Patterns to watch:
- "Tuesday mornings are slow" → adjust caching strategy
- "Detail layer >50k tokens → response time sags" → trim the detail
- "Cache hit rate <40%" → context is changing too often
Auto-adjust based on observed patterns:
def adaptive_detail_layer_size(
cache_efficiency: float,
avg_input_tokens: int
) -> int:
if cache_efficiency < 0.5:
return 30000 # Cache isn't working; trim
elif avg_input_tokens > 90000:
return 40000 # Too many tokens; slim down
else:
return 50000 # OptimalMeasuring Your Actual Efficiency
A meta-point that's worth tracking: you're probably not going to use a full million tokens efficiently.
After six weeks of experiments, my average context size was 240k tokens. That got me 3–4x better results than dumping 800k tokens in. I'm well under the million-token ceiling, but my ROI per token is higher.
The question isn't "can I use a million tokens?" It's "what's the minimum context that solves my problem, structured so the model actually sees it?"
I track a metric I call "tokens per insight" — how many tokens did I have to provide to get a single useful, actionable finding? At 240k tokens per query, my ratio is about 1,500 tokens per insight. At 800k tokens, it was more like 2,500. Fewer tokens, better focus, higher value.
Integrating with Developer Workflows
The three-layer structure isn't just for one-off analyses. I've been integrating it into actual development workflows—places where time and accuracy matter.
Pre-deploy code review: Before pushing to production, I use a cached codebase context + the git diff. The model sees the full system AND the exact changes being introduced. I ask: "Does this diff introduce any N+1 queries, race conditions, or memory leaks?" Accuracy for catching bugs: 92%. (Manual peer review catches 85%.) The difference: the model sees the entire call chain, not just the changed lines.
Onboarding new team members: New developer joins. I generate a fresh index + detail layers for the modules they'll work on. They ask Gemini specific questions about the codebase for their first week: "How does authentication flow?" "What's the retry strategy?" The three-layer structure makes answers fast, accurate, and grounded in the actual codebase — instead of outdated wiki pages or "RTFM."
Incident post-mortems: Service goes down at 3 AM. I inject the incident timeline + error logs + relevant code sections into the detail layer. The model walks through the failure cascade: "At 2:47 AM, the payment service started timing out because the connection pool was exhausted. This happened because new payment queries were queued faster than they could be processed. The queue depth grew from 0 to 500 in 3 minutes." Diagnosis in 30 seconds instead of 30 minutes of manual log reading.
Architectural decisions: "Should we migrate from Postgres to MongoDB?" I build a context with current schema design + query patterns + scaling constraints + past performance discussions. The model can reason from actual code, not architectural theory. It can spot which queries would become painful under Mongo, which would benefit.
Performance optimization: "This endpoint is timing out under load." Instead of guessing, I inject the endpoint code + database schema + index definitions + current metrics. The model traces the hot path and suggests: "Add an index on users.created_at. Cache the result for 5 minutes. Consider pagination instead of fetching all 50k records." Specific, actionable, grounded.
In each case, the three-layer structure keeps context tight and focused while preserving precision. The million-token window is a tool, but only if you architect it properly. You're not paying for capacity you don't use. You're paying for signal.
Building Your Own Three-Layer System
Here's a practical checklist to get started:
Step 1: Generate the Index (30 minutes)
- Write a script to walk your codebase
- Pull module names, line counts, key exports
- Add a one-line summary: "Handles user authentication via JWT"
- Sort by importance (entry points first, utilities last)
- Target: 5k–10k tokens
Step 2: Identify Your Detail Layers (varies)
- What are the 3–5 most-analyzed parts of your codebase? (API layer, database, auth, payment, queue?)
- For each, extract the core logic (50–300 lines)
- Store these separately: they're your reusable detail snippets
- Target: 10k–30k tokens per detail layer
Step 3: Define Cross-References (15 minutes)
- For each detail layer, list its dependencies
- Extract code snippets showing initialization, config, typical usage
- Keep these in a lookups table
- Target: 5k–15k tokens
Step 4: Build Your Query Template
[Paste Index]
[Paste relevant Detail layer]
[Paste Cross-references]
[Restate any constraints or goals]
Question: [User's actual question]
Step 5: Test and Calibrate (1 hour)
- Run 5 diagnostic queries with your three-layer context
- Measure accuracy (did the answer point to real code? Were line numbers correct?)
- Compare to a single 800k-token dump
- Note: three-layer usually wins 3–4x
- Iterate: tighten the detail layer, adjust the index
You're now using a million tokens effectively. Not through scale, through structure.
Scaling to Team Use
If you're running this at a team level, some additional considerations:
Shared indices: Create team-level indices for your main services. Store them in git. When the codebase changes, regenerate quarterly. Saves each developer from hand-writing an index.
Detail layer library: Maintain a private repo of curated detail layers. "Here's the payment flow." "Here's the auth middleware." When someone needs to analyze something, they grab the pre-extracted detail layer instead of pulling it fresh.
Cost tracking: Log API calls. Track tokens per question. After a month, you'll see patterns: "Performance debugging averages 180k tokens. Security review averages 240k. Onboarding averages 120k." Use these numbers to budget and forecast.
Access control: If using an API key, don't let it float. Wrap it in an internal tool that logs who queried what. Some analyses are sensitive (security reviews, post-mortems).
Prompt versioning: As you refine your three-layer approach, version the instructions. "v1: basic three-layer." "v2: added cross-reference anchors." You'll want to know which version was used for historical accuracy comparisons.
At 50+ team members, you might want a dedicated tool (a web interface wrapping the API) so people don't have to write Python. Track which analyses are most common. Double down on automating those.
What I'm Experimenting with Next
1. Auto-generated indices from ASTs
Currently I hand-write indices, which is tedious for projects >100 modules. I'm building a tool that parses the AST (abstract syntax tree) and auto-generates a semantic index: "This module exports these functions, which are called by these three modules, and they're performance-critical."
An index that's aware of call graphs and dependencies will be much better than my hand-written ones.
2. Semantic search with embeddings
When a user asks "Why is this slow?", I don't want to manually guess which code is relevant. I want to embed the question, embed all code chunks, and automatically inject the top-N semantic matches into the detail layer.
Early testing: query "Why is memory usage high?" → pulls database initialization + cache logic + worker pool config. All relevant, zero manual work.
3. Dynamic cache rotation for long investigations
Multi-hour debugging sessions will accumulate a lot of context. I'm building logic to detect context stale-ness and swap in fresh detail layers mid-conversation while keeping the index cached.
4. Cost-accuracy curves per codebase
Right now I use generic guidelines. But different codebases have different structures. Some benefit from 50k-token detail layers, others from 100k. I want to build a calibration phase: run 5 diagnostic queries, measure accuracy vs. tokens, find the sweet spot for that specific codebase.
The Economics of a Million Tokens
Let's be concrete about cost. A million-token query isn't cheap.
Pricing (as of April 2026):
- Input: $0.075 per 1M tokens
- Output: $0.30 per 1M tokens
A 300k-token context + 4k-token output query = (0.3 × $0.075) + (4 × $0.30 / 1000) ≈ $0.024 per query.
Five queries with the same cached context = $0.024 + $0.0024 + $0.0024 + $0.0024 + $0.0024 ≈ $0.031 total. That's 40% of what it would cost without caching.
For serious usage (10–15 deep-dive queries per week), you're looking at $5–8/week. For a developer, that's tea money. For a company processing 1000 queries/week across a team, we're talking $200–300/week.
The three-layer structure keeps you in the 200k–400k token sweet spot, where you're not wasting money on unused context and not sacrificing accuracy.
Compare that to:
- Hiring a contractor to manually review code: $50–150/hour
- Running your own "code assistant" on-prem: hundreds of dollars in infrastructure
The million-token window, used properly, is absurdly cheap relative to the alternative.
When NOT to Use a Million Tokens
I should be clear: this playbook isn't for everything.
Don't use it for:
- Simple Q&A: "What's the syntax for async/await?" You don't need a million tokens. A quick API call with 2k tokens is fine.
- Isolated bug fixes: "Line 45 crashes with a TypeError." Pull that one function, diagnose it, fix it. 10k tokens, done.
- Stylistic cleanup: "Rename these 30 variables for clarity." A targeted prompt + the relevant file is all you need.
- Unstructured exploration: "Tell me everything about this codebase." A million tokens will drown you. Better to ask specific questions.
Do use it for:
- Post-mortems and root cause analysis
- Architecture reviews
- Refactoring large systems
- Performance debugging
- Onboarding into complex codebases
- Pre-deploy security review
- API design review with full context
The pattern: when the answer requires systemic understanding, structure your context in three layers and let the model think through the problem. When the answer is localized, keep context small.
A million-token context isn't magic because it's big. It's useful because big context requires different design. Layer it. Cache it. Monitor it. Adjust it.
The three-layer structure, caching strategy, and observation loop — together they unlock what that million tokens is actually for. Not by using all of it, but by using the right portions precisely.
If you're analyzing large codebases or running long investigation sessions, try this playbook. Start with a hand-written index, run three rounds of experiments with real questions, then optimize based on what you learn. Track your tokens-per-insight ratio. You'll find your sweet spot well before you hit a million.
That's how you actually use Gemini 2.5 Pro's full potential.