How to reduce LLM API costs: twelve levers, ranked by risk

Most advice about reducing LLM costs starts and ends with “use a cheaper model”. That is one lever out of twelve, it carries the most quality risk of any of them, and in a large number of production systems it is not even the biggest saving available. Reaching for it first is how teams end up with a smaller invoice, a worse product, and no way to tell which change caused which.

This guide lists the levers in the order we apply them on client engagements: strictly increasing quality risk. Everything in the first group is provably output-identical — it cannot change what your customers see, so it needs no evaluation harness and no negotiation. The middle group changes outputs in ways you can measure. The last group changes them in ways you have to judge. Working in that order means you bank the free money before you spend anyone's political capital on a trade-off.

Group one: changes that cannot alter your output

These four are, by construction, output-preserving. The model receives exactly the same instruction and produces exactly the same class of answer; only the billing changes. They need no golden dataset and no sign-off from a product owner, which makes them the fastest work in any engagement.

1. Prompt caching — stop paying full price for tokens the provider has already read

Every major provider now discounts input tokens it has seen before, and the discount is steep — cached input typically costs a fraction of standard input, with the exact ratio varying by provider and changing over time. The catch is that caching is prefix-based: the repeated content must sit at the very front of the prompt and be byte-for-byte identical between calls.

This is where most systems lose the benefit by accident. A prompt assembled as “system instructions, then the current date, then the retrieved documents, then the user question” invalidates its own cache on every single call, because the date changes and everything after it is therefore new. Moving volatile content to the end — instructions, then documents, then date, then question — costs nothing, changes nothing about the answer, and can convert the majority of your input tokens into cached reads.

• Order prompts from most stable to least stable: system rules, tool definitions and few-shot examples first; retrieved context next; user turn last.
• Keep the stable prefix genuinely stable — no timestamps, no request IDs, no user names, no randomised example ordering inside it.
• Check that your framework is not silently reserialising the prefix. JSON key ordering, whitespace changes and re-templating all break byte-identity.
• Watch cache hit rate as a first-class metric. If it is not on a dashboard, it will regress the first time someone edits a prompt.

2. Batch endpoints — pay less for work nobody is waiting for

Providers discount asynchronous batch processing substantially against interactive rates, in exchange for a completion window measured in hours rather than seconds. An enormous amount of production AI work has no human waiting on it and is nevertheless billed at interactive prices out of pure habit.

• Overnight enrichment, classification and tagging backfills
• Document summarisation queues where the summary is read the next morning
• Evaluation and regression runs in CI, which are often a surprisingly large line item on their own
• Embedding generation for bulk ingestion
• Report and digest generation on a schedule

The engineering cost is a queue and a callback rather than a synchronous call. That is a day or two of work for a discount that applies for the life of the workload.

3. Deduplicate identical calls

Exact-repeat requests are more common than teams expect: the same document summarised by three users, the same FAQ asked forty times an hour, the same page re-analysed on every render because a component remounted. A response cache keyed on the full normalised request handles all of it, and for deterministic settings the cached answer is indistinguishable from a fresh one because it is the same answer.

The subtle version of this lever is idempotency. Retries triggered by timeouts frequently produce a second billed generation for a request the provider already completed and the client simply stopped listening to. Correct timeout handling and idempotency keys turn that double-charge into a single one.

4. Stop paying for output you discard

Output tokens are typically the most expensive tokens on the bill, several times the price of input. Any system that generates long text and then truncates it, parses one field out of it, or throws it away when the user navigates is burning the most expensive resource it has.

• Set max output tokens deliberately per call site rather than leaving a global default that is ten times what the feature needs.
• Use structured outputs or tool calls when you only want fields — a JSON object with three keys is a fraction of the tokens of a paragraph containing the same three facts.
• Cancel streams properly when a user navigates away; an abandoned stream that keeps generating is billed in full.
• Ask for the answer, not the reasoning, when the reasoning is never read.

Group two: changes you can measure your way through

These four change what the model sees or how often it is called. They can affect output quality, but the effect is measurable against a fixed dataset, which means the decision is empirical rather than political. Build the golden set once and all four become routine.

5. Trim retrieved context to the point where accuracy stops improving

RAG systems are usually configured once, early, with a generous number of retrieved chunks chosen to make the demo work. That number is then never revisited, and every request thereafter pays for it. The right way to set it is to plot accuracy on your own hard cases against the number of chunks retrieved. The curve almost always flattens well before the configured value.

Better retrieval beats more retrieval. A reranking step that lets you send five excellent chunks instead of twenty mediocre ones usually improves accuracy and cuts input tokens at the same time — one of the few genuinely two-sided wins in this list.

6. Fix conversation history before it goes quadratic

A chat product that resends the entire conversation on every turn has a cost curve that grows with the square of session length. Turn twenty costs twenty times what turn one cost, for a user who feels like they are having one conversation. This is the single most common structural cost defect we find in chat and agent products.

Combine this with caching and the effect compounds: a stable summarised prefix is cacheable in a way that a constantly-growing raw transcript never is.

7. Cut the number of calls per task

Agent and pipeline architectures often make several model calls where one would do — a classify step, then a plan step, then an execute step, each with its own full system prompt. Sometimes that decomposition genuinely improves reliability. Frequently it is an artefact of how the feature was prototyped.

• Count calls per completed task and treat it as a headline metric alongside tokens.
• Look for chains where one step's only job is to decide which of two branches to take — that is often a cheap classifier's job, or a regex's.
• Merge steps that always run together and share context; you pay for that context once instead of twice.
• Cap agent loop iterations explicitly. An unbounded loop is an unbounded invoice.

8. Fail fast, and fail once

Failures are billed. A call that returns malformed JSON, gets rejected by a validator and is retried has cost you twice for one task. Systems with a 20% silent retry rate are paying a 20% tax that appears nowhere in any dashboard, because both attempts look like ordinary successful calls to the provider.

• Instrument retries explicitly, and attribute both attempts to the same task.
• Use constrained decoding or structured output modes so malformed responses stop happening rather than being retried.
• Validate inputs before spending a model call on them — an empty document does not need summarising.
• Set timeouts that reflect real latency distributions, so you are not abandoning and re-requesting work that was about to arrive.

Group three: changes that need a judgement

The last four levers change the answer your customer receives. Each can be enormous. None should be shipped on an assumption, and all of them need the evaluation harness that group two built.

9. Route each task class to the cheapest model that passes

This is the lever everyone reaches for first and the one that most needs evidence. The price gap between a flagship model and a small one in the same family is routinely an order of magnitude, and a large share of production traffic — classification, extraction, tagging, short rewrites, intent detection — sits comfortably inside a small model's competence.

10. Shorten and sharpen the prompt itself

System prompts accumulate. Every incident adds a clause, every edge case adds an example, and nothing is ever removed because nobody is confident about what each line is holding up. A prompt that has grown to two thousand tokens is billed on every single call, forever, and a meaningful fraction of it is usually inert.

With a golden set this becomes tractable: remove a section, run the evaluation, keep the removal if quality holds. It is unglamorous and it compounds, because the prompt is the one token cost that every request pays.

11. Consider fine-tuning or distillation for narrow, high-volume tasks

For a task that is narrow, stable and very high volume, a small fine-tuned model can match a large general model at a fraction of the inference cost, because the instruction and examples that were costing you tokens on every call are now baked into the weights.

Be realistic about the total cost: training runs, dataset construction, evaluation, and a maintenance burden every time the task definition shifts. The break-even is a volume calculation, and below it the honest answer is that fine-tuning is a more expensive way to do the same thing.

12. Buy capacity differently once demand is predictable

Provisioned or reserved throughput converts variable spend into fixed spend, which is a good trade at high steady utilisation and a bad one at low spiky utilisation, because you pay for the troughs. The break-even depends on your actual hourly traffic curve, not on a rule of thumb, and it should be modelled against measured demand rather than a forecast.

This lever comes last deliberately. Committing to capacity before optimising volume locks in your current inefficiency for the length of the commitment.

What this looks like in practice

A typical engagement finds that groups one and two account for a substantial share of the achievable saving and carry effectively no quality risk, while group three carries the largest single number and all of the risk. Teams that start at lever nine and work backwards tend to spend months arguing about quality before touching the free money. Teams that start at lever one usually have the audit paid for before the first difficult conversation.