Labs / Experiments · Updated 2026-09-11
Is the cheapest model actually cheaper?
A cheaper AI model cost less per step and more per finished job.
Running experimentBuilt and running, being proven.
- Task submitted
- Picked up
- Working (turns counted)
- Checkpoint saved, or cap hit
- Cost read from provider
- Result compared per completed task
What we asked and what we found
Every entry answers the same eight questions, in this order.
Question
When the same bounded, read-only task goes to a cheaper model, does the total cost of a finished result go down?
Why it matters
Every business that runs on AI pays per token, and the obvious saving is to send routine work to a cheaper model. We wanted to know whether that holds when the unit of value is a finished task.
State
Running experiment Built and running, being proven.
What we built or tested
- The same kind of work sent to two models through Lehvel's own task system.
- A cap of 40 turns and a required saved checkpoint, so an unfinished run counts as a clear failure.
- Cost read from the provider's bill, not estimated.
Evidence
- High-capability model
- 34 turns, finished, wrote a usable checkpoint, $2.47
- Cheaper model
- 41 turns, hit the cap, no usable checkpoint, $0.44
- Cost per turn
- About 6.7 times cheaper on the cheaper model
- Cost per finished task
- Undefined on the cheaper model. It never finished.
What we learned
- Cost per token or per turn is not cost per finished task. The cheaper run was cheaper and worthless.
- A read-only task can still need judgment the cheaper model lacked.
- Write routing rules per kind of work, from measured completions, not from list prices.
Limitations
- One kind of task, two models, a small number of runs. This is a finding about a pattern, not a benchmark.
- A different kind of task could reverse the result. That is the point of measuring each kind.
What changes next
Run the same test across the kinds of work we send every day and publish the routing table with its measured completions.
Failed tests are published as results. A state on this page describes today, not plans.
Next in Labs
- Next: Sales Desk
Our client sites run on checked facts, so our sales record does too. One system carries a customer from first conversation into delivery.
Systems
Get Upstream
One email a week: what we asked the AI, what it got wrong, and how we checked.
Our research reaches 17,000+ followers across our channels. Upstream is the weekly version.
Weekly. Unsubscribe anytime. Privacy