Cost per successful task: the only AI metric that survives a CFO

There is a specific and expensive failure mode in AI cost work. A team moves a workload to a cheaper model, the per-token cost falls by 80%, the dashboard turns green, and everybody moves on. Three weeks later support volume is up, users are rephrasing their questions twice as often, and the total bill has not moved — because the cheap model needs two attempts and one escalation to do what the expensive one did in a single call.

Nothing in a per-token view catches this. Every one of those extra calls is a perfectly ordinary, correctly-priced, successfully-served API request. The measurement was not wrong; it was measuring the wrong thing. It measured the cost of tokens when the business buys completed tasks.

The definition

Cost per successful task is all variable AI cost, including the cost of failures, divided by the number of customer tasks actually completed:

Two properties make this metric different from everything else on an AI dashboard. First, it is denominated in units of business — tasks, not tokens — so it can be compared against revenue per task, against a human baseline, or against last quarter without translation. Second, it punishes the exact failure mode above automatically. A cheaper model that needs more attempts moves this number in the wrong direction on the day it ships.

Defining “successful” is the hard part, and it is not a technical question

Teams get stuck here, and they get stuck because they look for the answer in the wrong department. A model call returning HTTP 200 is not success. A response passing a JSON schema is not success. Success is whatever your product promised the user, and only the product knows what that was.

The right definition is usually the one already sitting in a product metric somewhere: resolution rate, acceptance rate, conversion, completion. That is a feature, not a coincidence. If your AI success definition matches a metric the business already trusts, nobody has to be persuaded that the denominator is fair.

One warning: pick the definition before you start optimising, and write it down. A success definition chosen after the results are in is not a measurement, it is a negotiation.

Instrument it so no single service can flatter the number

The structural insight that makes this metric trustworthy is that neither half of it can be produced by the service that benefits from it looking good.

• The gateway — or whatever sits in the model call path — knows tokens, model, cached tokens, latency and price. It has no idea whether the customer was served.
• The application knows whether the lead was captured, the ticket was closed, the draft was kept. It has no idea what any of that cost.

Two independent writers, one shared join key — a task or request identifier that travels through both paths. Neither side can quietly improve the ratio, because neither side holds both halves of it. This separation is not bureaucratic caution; it is what makes the number defensible in a room where somebody's project is being judged by it.

What the number tells you that the invoice cannot

Once the join exists, several questions that were previously arguments become lookups.

How much are failures costing?

Split total spend into cost-on-successful-tasks and cost-on-failed-tasks. The second figure is usually the first genuinely uncomfortable number an audit produces, and it is frequently larger than any model-switching saving on the table. It is also spend with no defender — nobody has to be persuaded that money burned on tasks that failed is worth reducing.

Is this feature worth what it costs?

Cost per successful task compares directly to the value of that task, in the same units. Suppose an automated resolution costs a fraction of what the equivalent human ticket costs — that is an easy conversation. Suppose a lead-qualification flow costs more per captured lead than the lead is worth — that is a hard conversation, but it is at least a conversation the numbers can frame. The per-token view cannot pose either question, because it has no denominator the business recognises.

Which model actually wins?

This is the original question, and now it has an answer rather than an opinion. Run the candidate in shadow, compute cost per successful task for both configurations on the same traffic, and compare. A model that is cheaper per call and worse at the job loses on this metric, visibly, before it ever reaches a customer.

Common ways teams get it wrong

• Counting only the successful attempt's cost. If a task took three tries, all three belong in the numerator. Excluding the failures is how you accidentally prove that failing more often is cheaper.
• Letting the model judge its own success. Asking the same system whether it did well produces a metric that improves whenever the model becomes more confident, which is not the same as becoming more correct.
• Redefining success mid-flight. Any change to the definition resets the series. Version the definition and annotate the chart at the change, or the trend is fiction.
• Reporting one estate-wide average. Different features have wildly different economics; a single number hides both the disaster and the success story inside it.
• Ignoring non-AI costs that the AI causes. An answer that triggers a human escalation costs you a human. If escalations are the failure mode you are trying to reduce, price them.

Where it fits

Cost per successful task is not a replacement for per-token monitoring — you still need that to debug a spike at three in the morning. It is the layer above it: the number that decides whether a change was actually good, and the one that translates an engineering decision into a sentence a finance director can act on.

It is also, in our experience, the fastest way to end the recurring stalemate where engineering says the AI feature is efficient and finance says it is expensive. Both are usually right about different denominators.