Free money first
Prompt, context and caching engineering: send fewer tokens, get the same answer
The cheapest token is the one you never send. We restructure prompts for caching, trim context that adds cost but not accuracy, and move eligible work to batch — changes that are provably output-identical.
Outcome: A materially smaller bill from work that leaves the answers your customers see completely unchanged.
Talk about prompt, context & caching engineeringMost AI bills are volume problems wearing a pricing costume
Teams reach for a cheaper model first because model choice is the visible lever. But in a large share of production systems the dominant term is not the unit price — it is how many tokens travel on each request, and how many times the same tokens travel.
A chat product that resends the full conversation on every turn grows its own bill quadratically over a session. An agent loop that re-reads its instructions on every step pays for the same system prompt dozens of times. A RAG pipeline that retrieves twenty chunks to answer from three pays for seventeen every single time. None of that presents as a pricing problem, and all of it lands on the invoice.
The four moves, in the order we make them
Make the static part cacheable
Prompt caching charges a steep discount for input the provider has already seen, but only when the repeated content sits at the front of the prompt and is byte-identical. Most prompts are assembled in an order that defeats this by accident — a timestamp or a user name near the top invalidates everything after it. Reordering costs nothing and changes nothing about the output.
Stop paying for context that does not earn its place
We measure accuracy as retrieved context is trimmed, using your golden set, and find the point where accuracy stops improving. In most RAG systems that point arrives well before the number of chunks currently being sent. Everything past it is pure cost.
Fix the conversation history strategy
Full-history resends are the most common quadratic cost in chat products. Rolling summaries, windowed history and selective recall each trade a little fidelity for a lot of money, and the right choice differs per product. We test it rather than assert it.
Move what can wait to batch
Enrichment, classification backfills, nightly summarisation and evaluation runs are usually billed at interactive rates for no reason at all. Major providers discount asynchronous batch work substantially. Nobody is waiting for it, so nobody notices the latency.
Why this comes before model changes
These changes are, by construction, output-preserving or output-testable. Caching a system prompt cannot change an answer. Moving a nightly job to batch cannot change an answer. Trimming retrieval can, so we measure it — but against a fixed model, which makes the measurement clean.
That matters for sequencing. If you change the model and the context and the caching at once and the bill halves, you have learned nothing about which change to keep when quality complaints arrive. We take the free savings first, bank them, and only then spend evaluation effort on the trades that involve a real decision.
Provider list prices and cache discounts change frequently, so we reconcile every projection against your actual invoice before anyone reports a number upwards.
Deliverables
What you have at the end
- Token profile: where volume comes from, per feature and per turn
- Cache-aware prompt restructuring, with before/after hit rates
- Context and retrieval trimming, validated against your golden set
- Conversation-history strategy for multi-turn products
- Batch and off-peak scheduling for latency-insensitive workloads
- Guardrails so context creep is visible the next time it starts
Everything on that list lives in your repositories and your cloud account. Ending an engagement does not take the capability with it.
FAQ
Prompt, Context & Caching Engineering: common questions
How much does prompt caching actually save?
It depends entirely on what fraction of your input tokens are repeated and stable. A product with a long fixed system prompt and short user turns can see most of its input cost become cached reads at a fraction of the standard rate. A product where every request is unique gains nothing. The token profile in week one tells us which you are before we commit to a number.
Isn't trimming context just making the product worse quietly?
It would be, without measurement — which is why the golden set comes first. We trim only to the point where accuracy on your own hard cases is flat, and we show you the curve. If it bends, we stop.
Can you do this without touching our application code?
Partly. Caching, batching and routing changes can often live in the gateway. Prompt structure and retrieval strategy live in your application, so that part is a joint piece of work with your engineers — usually a few focused days rather than a rewrite.
Does this apply to agent workloads?
Especially to agent workloads. Multi-step agents re-establish context on every step, which makes them the most cache-sensitive and context-sensitive systems most teams run. They are frequently where the largest single saving in the estate is hiding.
Related
Where this leads next
AI Spend Audit
A two-week, fixed-fee forensic review of your AI bill. We instrument your calls, attribute every pound to a feature and an outcome, and return a ranked list of savings with the expected quality impact of each one.
Model Routing & Evaluation
The largest single lever in most AI bills is that one model serves every request. We segment your traffic by task, find the cheapest model that still passes, and prove it on your own traffic before it reaches a customer.
Prompt caching explained
Caching is prefix-based, which means one timestamp in the wrong place can cost you the entire discount. Here is how it actually works and how to structure prompts so it fires.
Find out what your AI actually costs per completed task.
Two weeks, a fixed fee, and a ranked savings plan with the quality risk of every move stated up front. If the numbers say an audit is not worth it for you, we will say so on the first call.