Model routing: how to use cheap models without shipping worse answers

Almost every AI product starts by pointing everything at the best model available. That is the correct first decision — you are trying to find out whether the idea works at all, and the fastest way to remove model quality as a variable is to remove it as a variable. The problem is that the decision then becomes permanent by default, and a year later a flagship model is doing work that a model costing an order of magnitude less would do identically.

Model routing is the practice of sending each class of request to the cheapest model that demonstrably handles it. The word carrying the weight there is “demonstrably”. Routing without evaluation is not cost optimisation; it is a quality change with a cost side effect, shipped on a hunch.

Start by segmenting traffic, not by comparing models

The instinct is to open a benchmark leaderboard. The useful first move is to look at your own traffic and group it by what it actually asks the model to do. A single product surface usually contains several genuinely different jobs with wildly different difficulty.

That table is a worked illustration rather than a measurement, but the shape of it is the point: the moment you have your own version, the strategy writes itself. Three quarters of the calls are cheap, easy work, and they are not where the money is. Answer generation is a fifth of calls and the majority of spend, and it is the one class where quality is the product. The classification traffic can move to a small model this week with a low ceiling on the saving; the answer generation needs real evaluation and carries the entire prize.

Build a golden dataset from your own hard cases

Public benchmarks tell you how models rank on somebody else's distribution. They cannot tell you whether a small model handles the specific phrasing your customers use, the specific documents you retrieve, or the specific edge cases that generate your support tickets. Only your traffic can.

A golden set is a representative sample of real requests per task class, with a reference judgement about what a good response looks like. Building one is the single most valuable artefact of a routing project, because everything afterwards — model changes, prompt edits, provider migrations, deprecation forced on you at short notice — is evaluated against it for years.

• Sample across the real distribution, not just the clean middle: include the very long inputs, the ambiguous ones and the multilingual ones if you have them.
• Deliberately over-sample known failures. The cases that produced complaints or escalations are the ones a cheaper model is most likely to fail differently on.
• De-identify. There is no reason for an evaluation corpus to hold personal data, and removing it early avoids an awkward conversation later.
• Record the judgement, not just the expected string. For open-ended generation, “what would make this answer acceptable” is more durable than one reference answer.
• Version it. When the set changes, the scores are not comparable, and you will want to know which version produced which number.

Replay candidates in shadow before anyone is affected

Shadow replay means running candidate configurations against real traffic offline. The customer receives the production answer as always; the candidate's answer is scored and discarded. It removes the entire class of risk that makes routing projects stall, because nothing you are testing can reach anybody.

On scoring: an LLM judge is fast and cheap, and it is trustworthy exactly to the extent that it has been calibrated against human labels on your data. Score a few hundred examples by hand, check the judge agrees, and then use it at scale. An uncalibrated judge is an efficient way to confirm whatever you already believed.

Gate the change so the saving does not decay

A routing decision that is correct today degrades quietly. Someone adds three sentences to a prompt after an incident. A provider deprecates the model version you validated. Traffic shifts as the product finds a new audience. None of that announces itself.

The countermeasure is an evaluation gate in CI: the golden sets run on every change that touches a prompt, a model choice or a routing policy, and a regression below an agreed threshold fails the build. This is the difference between a cost project and a cost practice. Without it, most of the saving is gone within two quarters and nobody can say when it went.

• Set the threshold explicitly and agree it with the product owner, in writing, before the first ramp.
• Run the gate on prompt changes as well as model changes — prompts regress quality far more often than models do.
• Keep the routing policy versioned and separate from application code, so a rollback is a config revert rather than a deploy.
• Alert on cost per successful task in production, because the gate protects quality on your sample and production always finds inputs your sample did not.

Roll out as a ramp with a rollback

Move a small percentage of live traffic first, watch for a period that includes at least one full daily cycle, then increase. Through the ramp, the metric to watch is cost per successful task rather than per-token cost, because the failure mode you are guarding against — a cheaper model that needs more attempts — is invisible in the token view and immediate in the ratio.

Keep the rollback to a single configuration change, and test it before you need it. A rollback path that requires a deploy is a rollback path you will hesitate over at exactly the wrong moment.

When not to route

Three cases where the honest answer is to leave it alone.

• The class is a small share of spend. Evaluation effort is real; spending four weeks to save 3% of the bill is a bad trade against spending it on the class that is 60%.
• The consequences of a wrong answer are legal, clinical or financial. Where an error is not just annoying, the correct routing decision is often to keep the strongest model and take the saving from caching, context and batching instead.
• You have no way to define success for the class. If you cannot say what a good answer is, you cannot evaluate a candidate, and routing becomes guessing with extra steps. Fix the definition first.