Prompt caching explained: how to stop paying twice for the same tokens

Prompt caching is the highest-return, lowest-risk change available in most AI systems, and it is also the one most often left half-working. The discount on repeated input is steep across every major provider. The requirement for earning it is unforgiving in a way that catches nearly everybody the first time: the repeated content has to be a byte-identical prefix.

This guide covers what that means mechanically, the four ordering mistakes that disable caching by accident, and how to tell whether yours is working.

What providers are actually charging you for

When a model processes your prompt, it computes an internal representation of every token — the attention key/value state. That computation is a large share of what input tokens cost. If two requests begin with exactly the same tokens, the state for that shared beginning is identical, so it can be computed once and reused.

That is the whole trick, and it explains every rule that follows. Reuse is possible only from the start of the prompt and only up to the first token that differs, because everything after a change has different context and must be recomputed. This is why caching is described as prefix-based rather than as a lookup table: there is no partial credit for repeated content in the middle of a prompt.

Commercial terms differ by provider and change over time: some apply caching automatically above a minimum prefix length, others require an explicit marker in the request; some charge a small premium to write a cache entry in exchange for a much larger discount on reads; retention windows range from minutes to considerably longer, usually extended by reuse. Check your provider's current documentation for the numbers — but the mechanism above is universal, and it is the mechanism that dictates how you write the prompt.

The four ways teams disable caching by accident

1. Volatile content near the top

This is the big one, and it is almost always unintentional. A prompt assembled as “you are a helpful assistant… / today's date is 30 August 2026 / the user is Sarah Chen / here are the retrieved documents… / here is the question” looks perfectly sensible and caches nothing beyond the first line, because the date and the name change and everything after them is therefore new on every request.

The fix costs nothing and changes nothing about the output. Order the prompt from most stable to least stable:

1. 1System instructions, role, rules, output format — identical for every user, forever.
2. 2Tool and function definitions — stable until you ship a change.
3. 3Few-shot examples — stable, and worth keeping in a fixed order.
4. 4Long shared reference material — a policy document, a schema, a codebase file.
5. 5Per-user or per-session context — profile, permissions, conversation summary.
6. 6Retrieved context for this request.
7. 7The volatile scraps: current date, request ID, timestamps.
8. 8The user's actual question, last.

If the model genuinely needs today's date to answer well, it does not need it in line two. Putting it immediately before the question works just as well and preserves everything above it.

2. Byte-identity broken by serialisation

“Identical” means identical bytes, not identical meaning. Several common patterns break this without any visible change to the prompt:

• JSON serialised with non-deterministic key ordering — the same object, a different byte string.
• Trailing whitespace or line-ending differences introduced by templating.
• Few-shot examples shuffled per request to “avoid position bias”, which also shuffles away the cache.
• A framework that rebuilds the system message with a fresh UUID, trace ID or timestamp header.
• Prompt fragments interpolated from a set with non-deterministic iteration order.

The diagnostic is straightforward: log the exact serialised prefix for two consecutive requests and diff them. Teams are routinely surprised by what shows up.

3. A prefix shorter than the provider's minimum

Providers impose a minimum cacheable prefix length. Below it, nothing is cached however stable the content is. Systems with very short system prompts and long user inputs sometimes sit just under the threshold and see no benefit at all — occasionally worth restructuring toward, if there is genuinely stable material that could be promoted into the prefix.

4. Traffic too sparse to keep the cache warm

Cache entries expire. A workload with one request every twenty minutes may never hit a warm cache, while the same prompt structure under continuous traffic hits it almost always. This is why the same code can show excellent cache rates in production and none at all in a staging environment, and why batching related work together in time can be worth doing for cost reasons alone.

Designing for cacheability from the start

Once the prefix rule is internalised, some architectural choices become obvious.

The conversation-history case is worth dwelling on because it compounds with the quadratic cost problem. Replacing a growing raw transcript with a summary plus the last few turns does two things at once: it stops the input growing without bound, and it makes the earlier part of the prompt stable enough to cache. The two effects multiply rather than add.

How to tell whether it is working

Providers report cached input tokens separately in the usage object on each response. That field is the ground truth, and it belongs on a dashboard.

• Track cache hit rate as cached input tokens divided by total input tokens, per feature rather than estate-wide.
• Alert on a drop. A prompt edit that reorders the template will show up here immediately and nowhere else.
• Assert on it in tests where it matters: a test that fails when the stable prefix stops being stable is cheap to write and catches the regression at review time.
• Compare cost per successful task before and after, not just the token counts — caching changes the bill without changing the answers, so the ratio should move cleanly.