The big lever
Model routing and evaluation: cheaper models, proven not assumed
The largest single lever in most AI bills is that one model serves every request. We segment your traffic by task, find the cheapest model that still passes, and prove it on your own traffic before it reaches a customer.
Outcome: The cheapest model that demonstrably does the job for each class of request — with the evidence to defend the change.
Talk about model routing & evaluationOne model answering everything is the default, and it is expensive
Most AI features are built by pointing everything at the best model available, because that is the fastest way to make the thing work at all. It is a completely reasonable first decision and a costly permanent one. The price gap between a flagship model and a small one in the same family is routinely an order of magnitude per token, and a meaningful share of production traffic is doing work a small model handles perfectly.
Classification, routing, extraction, tagging, short rewrites, intent detection and “did the user mean X or Y” all tend to fall well inside a small model's competence. Long-form reasoning, nuanced drafting and anything with legal or clinical consequences usually does not. The value is in knowing precisely where the line sits for your workload rather than guessing at it.
How we find the line without breaking anything
Segment the traffic
Real requests get grouped into task classes by what they ask the model to do. Each class gets its volume, its share of spend, and its tolerance for error stated explicitly. A class that is 40% of calls and 4% of cost is not where you start.
Build a golden set from reality
For each class we assemble a representative dataset from your own traffic, including the awkward cases — the long ones, the ambiguous ones, the ones that failed. Public benchmarks measure the wrong thing; your hard cases are the only ones that matter.
Replay in shadow
Candidate configurations run against that traffic offline. No customer sees the output. Every candidate is scored on quality against the golden set, on cost at your real token distribution, and on latency. The comparison is like-for-like because it is the same requests.
Gate the change
The winning policy ships behind an evaluation gate in CI. If a later change to a prompt, a model version or a routing rule regresses the golden set, the deploy fails. This is what keeps a saving from quietly decaying three months after the invoice went down.
Ramp, watch, roll back
Rollout is a percentage, not a switch. We watch cost per successful task — not per-token cost — through the ramp, because a cheaper model that raises the retry rate shows up in that number immediately and in the token dashboard never.
The number that decides it
Every routing decision is judged on one measurement: all variable AI cost, including the cost of failures, divided by the number of customer tasks actually completed. A model that is 90% cheaper per call and needs two attempts and one human escalation is not a saving, and this is the only common metric that says so out loud.
It is also the number that survives a conversation with a CFO, because it is denominated in units of business rather than units of tokenisation.
Deliverables
What you have at the end
- Traffic segmentation into task classes with volume and cost per class
- A golden evaluation dataset per class, built from your real traffic
- Shadow replay results: candidate models scored on quality, cost and latency
- A versioned routing policy with per-class model assignment and fallbacks
- An evaluation gate wired into CI that blocks quality regressions
- Staged rollout — canary, percentage ramp, and a one-line rollback
Everything on that list lives in your repositories and your cloud account. Ending an engagement does not take the capability with it.
FAQ
Model Routing & Evaluation: common questions
Will a cheaper model make our product worse?
For some classes of request, yes — and those keep the expensive model. The point of the evaluation harness is that the answer stops being a matter of opinion. We can show, per task class, exactly where quality falls off and by how much, on your traffic.
How large does a golden dataset need to be?
Smaller than people fear, and it depends on how varied the class is. A tightly-scoped classification task can be characterised well with a few hundred labelled examples. Open-ended generation needs more, and needs human judgement in the loop. We size it during the audit rather than committing in advance.
Do you use an LLM to judge the outputs?
Where the task allows it and the judge has been calibrated against human labels on your data, yes — it is fast and cheap. Where it does not, we say so and use human review for the sample that matters. An uncalibrated LLM judge is a good way to prove whatever you already believed.
What stops the saving from eroding over time?
The evaluation gate in CI and the cost-per-success metric on the dashboard. Erosion is normal — prompts grow, models get deprecated, traffic mix shifts — so the work is designed to detect it rather than assume it away.
Related
Where this leads next
AI Spend Audit
A two-week, fixed-fee forensic review of your AI bill. We instrument your calls, attribute every pound to a feature and an outcome, and return a ranked list of savings with the expected quality impact of each one.
LLM Gateway Implementation
The single OpenAI-compatible endpoint every AI call in your estate goes through. It routes, retries, caches, enforces budgets and records what everything cost — deployed in your own cloud, not ours.
Prompt, Context & Caching Engineering
The cheapest token is the one you never send. We restructure prompts for caching, trim context that adds cost but not accuracy, and move eligible work to batch — changes that are provably output-identical.
Find out what your AI actually costs per completed task.
Two weeks, a fixed fee, and a ranked savings plan with the quality risk of every move stated up front. If the numbers say an audit is not worth it for you, we will say so on the first call.