Labs / Experiments · Updated 2026-09-17
What if the expensive AI is doing the wrong job?
We are testing whether a small AI model can handle the many small judgments around big-model work, without being allowed to act.
Running experimentBuilt and running, being proven.
- Real work
- Simple rules first
- Small-model judgment
- Confidence check
- Existing model or person
- Final outcome
- Compare and measure
What we asked and what we found
Every entry answers the same eight questions, in this order.
Question
Can a small decision model cut expensive model calls, the amount of material sent to a model and human review, by handling narrow sorting, routing, scoring and checking around real Lehvel work?
Why it matters
Most AI systems use their most powerful model for two jobs at once: solving hard problems and making small judgments around them. Is this evidence strong enough? Is this case routine or unclear? We want to know whether splitting those jobs makes the whole system faster, cheaper and more reliable.
State
Running experiment Built and running, being proven.
What we built or tested
- A small, bounded checking step around work Lehvel already does. It is not a new agent system.
- A division of labor: the small model judges, routes, scores and checks; ordinary code enforces the rules; large models write and reason.
- Side-by-side tests against real task records, completion reports and research work before any shortcut reaches production.
- Fixed sets of questions, with measurements of confidence, speed, cost, human corrections and confident mistakes.
Evidence
- What it does
- Takes the state of a task and returns a choice, a score or a yes-or-no, with probabilities. It writes no prose.
- Published input price
- About $0.04 per million input tokens through Vercel's AI Gateway, as of 2026-09-17
- Lehvel result
- Not measured yet. Testing on past and parallel cases is starting.
What we learned
- The opportunity is not replacing Claude or OpenAI. It is stopping large models from making every small judgment around the work.
- A neatly formatted answer is not a true answer. A confident decision can still be wrong, so confidence never becomes permission.
- Running it behind the gateway we already use keeps it replaceable and avoids turning a model test into a new stack.
Limitations
- TypeSafe and Vercel benchmark claims are vendor results, not Lehvel results. We will not repeat them as our performance.
- No Lehvel workload has yet earned a production shortcut from it. For now it runs only on the side.
- It may advise on routing or checking. It cannot choose a client, grant permission, publish, contact a customer, change a public page, move money or override a fixed rule.
- If it adds a step without removing meaningful cost, delay or rework, we drop it.
What changes next
Run it on real task records and publish how often it agreed with people, where it was confidently wrong, how fast it was, what it cost and how much large-model work it avoided. Keep what earns its place and delete the rest.
Failed tests are published as results. A state on this page describes today, not plans.
Next in Labs
- Next: Is the cheapest model actually cheaper?
A cheaper AI model cost less per step and more per finished job.
Experiments
Get Upstream
One email a week: what we asked the AI, what it got wrong, and how we checked.
Our research reaches 17,000+ followers across our channels. Upstream is the weekly version.
Weekly. Unsubscribe anytime. Privacy