Is the cheapest model actually cheaper?
Lower cost per turn did not mean lower cost per completed task.
RUNNING LABAn experiment actively being exercised and measured. Results may change.
01 Task submitted02 Claimed by lane03 Working (turns counted)04 Checkpoint written — or cap hit05 Billing read from provider06 Result compared per completed task
01 · Question
When the same bounded, read-only task is routed to a cheaper model, does the total cost of a finished result go down?
02 · Why it matters
Every AI-operated business pays per token. The obvious cost control is to route routine work to cheaper models. We wanted to know whether that obvious move holds when the unit of value is a completed task, not a completed turn.
03 · State
RUNNING LAB An experiment actively being exercised and measured. Results may change.
04 · What we built or tested
- The same class of work dispatched through our coordination plane to two models.
- A 40-turn cap and a checkpoint requirement, so an unfinished run is a measurable failure rather than a vague one.
- Per-run billing captured from the provider, not estimated.
05 · Evidence
High-capability model34 turns · finished · wrote a usable checkpoint · $2.47
Cheaper model41 turns · hit the cap · no usable checkpoint · $0.44
Cost per turn≈ 6.7× cheaper on the cheaper model
Cost per completed taskUndefined on the cheaper model — it never completed
06 · What we learned
- Cost per token or per turn is not cost per completed task. The cheaper run was cheaper and worthless.
- Tool permissions do not tell you how much judgment a job needs. A read-only task can still require judgment the cheaper model lacked.
- Routing rules should be written per class of work, from measured completions, not from list prices.
07 · Limitations
- One task class, two models, a small number of runs. This is a finding about a pattern, not a benchmark.
- A different task class could reverse the result — that is the point of measuring per class.
08 · What changes next
Extend the same harness across the work classes we actually dispatch daily and publish the routing table with its measured completions.
Negative results are published as results. A state on this page is a claim about what is true today, not about what is planned; a limitation is a limitation.
Tell us what is true about your business.
A few questions, in your own words — type them or talk. A person reads what you send and follows up. Nothing is published and nothing is sent on your behalf.