When Context Caching Didn't Lower My Gemini Bill — Field Notes on Measuring the Real Hit Rate

Standing up a single hit-rate meter

One manual look can't reveal what "sometimes" really is. I slipped a meter into the request path that keeps the breakdown of every call and aggregates it into a hit rate. You don't need a heavy dashboard — one structured log line is enough.

def log_cache_stats(resp, sink):
    u = resp.usage_metadata
    cached = u.cached_content_token_count or 0
    prompt = u.prompt_token_count or 0
    sink.emit({
        "cached_tokens": cached,
        "prompt_tokens": prompt,
        # token-weighted ratio: what share of input came from cache this call
        "cache_ratio": (cached / prompt) if prompt else 0.0,
        "hit": 1 if cached > 0 else 0,
        "output_tokens": u.candidates_token_count or 0,
    })

Two aggregates matter. One is the hit rate (the mean of hit — how often the cache took effect at all). The other is the token-weighted effective hit rate (sum of cached_tokens divided by sum of prompt_tokens). The first captures the frequency of "hit or miss"; the second captures how much the cache actually moved cost.

My hit rate was 78% — a respectable number. But the token-weighted effective rate was only 34%. That gap was the whole mystery. The cache was working on most requests yet not touching cost, because frequent rewrites were eating back the savings.

Three silent causes of a bill that won't fall

Once the meter filled in, the causes narrowed to three. None of them throws an error, so each stays silent until you measure it. In production, this "loss that never errors" is the most dangerous kind.

The first is TTL churn. A cache entry expires, and the next call rewrites it. The write price is roughly the same as normal input — far higher than the read price (about a quarter of input). If access is sparse but the TTL is short, you repeat "create, barely read, expire, create again," and only the write cost accumulates. My TTL was 3,600 seconds, but overnight the gaps between requests exceeded it, so I was rebuilding the entry on nearly every call. The fix comes later.

The second is per-user fragmentation. My first implementation created a cache entry per user. The system prompt is identical for everyone, yet there was one entry per person. Write count scaled with user count, each entry got too few reads to pay itself back, and most expired first. The fix was to have a single shared entry for the common part, used by everyone.

The third is leaning on implicit caching. Gemini 2.5 models apply implicit caching automatically when prefixes match, but it won't fire below a minimum token threshold (roughly 1,000–2,000 tokens depending on the model) or when the leading content changes per request. A few paths where I had dropped explicit caching "because implicit would handle it" were, in fact, passing through at full price.

Putting a number on the break-even point

Whether TTL churn helps or hurts is a break-even question, not a feeling. It comes down to how many hits inside one TTL window are needed to recover the write cost.

I express prices as ratios. Set normal input to 1, cache read to 0.25, and a write to 1.0 (one-time). For an 8,000-token system prompt, one hit saves 8,000 × (1 − 0.25) = 6,000 input units, while a write costs 8,000 units. So the hits needed to break even are 8,000 ÷ 6,000 ≈ 1.34. Unless you average more than about 1.4 hits within a TTL window, adding the cache makes you spend more, not less.

Laying out relative cost by hits-per-window (cache-less = 100) makes the call obvious at a glance.

The moment I drew this table, the overnight losses jumped out. Time windows with only one hit were consuming the savings. The fix wasn't "extend the TTL" — it was "match the TTL to call density": shorter during the busy daytime, and skip the cache entirely overnight, sending plain input instead.

Tracking average cost and margin daily

Whether the hit rate is truly fixed shows up, in the end, in the average cost per request (ACR). I convert reads, writes, and output into unit prices, compute the effective cost of one request, and track it daily. Watching the hit-rate chart alone lulls you into missing a swelling write cost.

Across the whole service, I derive gross margin from monthly cost per user divided by the monthly price. What matters here is not the average but the cost share of the top 5% of heavy users. Once that group starts taking more than 30% of total cost, you are past the point where hit rate is the issue — you are in the territory of designing a higher tier or a usage cap for them. Cost optimization and pricing always become the same road eventually.

Tightening it one step at a time

Rebuilding everything at once hides which change actually helped. I applied one move at a time and watched the ACR after each.

First, I split the system prompt into fixed and dynamic parts and moved the fixed part into a single shared cache. Per-user data now only gets appended to contents, fixing the entry count at one. This alone cut write count to a fraction and sent the effective hit rate climbing.

Next, I matched the TTL to call density. I dropped the fixed 3,600 seconds, estimated expected hits per window from the recent average request interval, and added a branch that skips the cache during windows likely to fall below 1.4 hits.

Finally, I hunted down the paths that had been leaning on implicit caching — anything still flowing through with cached_content_token_count at 0 — and moved them to explicit caches. Short prompts that fall below the threshold I removed from caching entirely, reaching instead for a model swap to Flash or another lever. In that case, optimizing the output side pays off more directly than caching.

Keeping the decision to not cache

Measuring taught me that Context Caching is not universal. It fits poorly with services whose system prompt is under 1,000 tokens, personalization types where the entire premise changes per request, and one-off tools used only a few times a month. In each, the window closes before the write cost is recovered.

For those, rather than forcing a cache, it's cleaner to open a different tap: tightening the output format to cut output tokens, using structured output to prevent double-billing from retries, or mixing in the Flash model. Caching turned out to be not "cache and win" but "cache when density clears the break-even point."

The next move

If you've enabled Context Caching but the bill hasn't dropped as much as you expected, before adding a single dashboard, just print cached_content_token_count for one request. If it's 0, you're passing through on implicit caching; if it's smaller than expected, it's fragmentation or a threshold miss; if the hit rate is high but cost won't fall, it's TTL churn. There are only three causes, and the triage starts from one metadata field.

I'm still tuning how I match TTL to density myself. If this gives you a starting point for the same triage, I'll be glad. Thank you for reading.