Why we looked around the model, not at it
Most weeks the AI news is a model announcement, and for a company that has to ship it is the least useful item in the feed. We already have frontier models. What we lack is a settled answer to the questions surrounding them: what context should a run see, how do we know the answer was any good, what is allowed to act on it, and how do we prove afterward that it happened.
Every item below answers one of those, and we did not go looking for that pattern. Not one of the seven is a smarter model. We also label what we tested separately from what we merely read, because conflating the two is how a research note becomes a marketing document.
The remaining wins are around the model, not inside it.
TypeSafe’s Jev — a judgment primitive, not another chatbot
Jev does not write prose. It takes program state and returns typed probabilistic decisions — booleans, choices, scores, with probabilities attached — in one parallel pass. What caught our attention was not the speed or price, but that the shape matches a job we were already doing badly.
The useful model is: state in, typed probabilistic judgment out, deterministic code decides what happens next. Our systems make hundreds of small semantic judgments around the real work. Is this evidence strong enough to support a recommendation? Is this execution report verified, or merely success-shaped? We were spending frontier-model cycles on every one — rather like hiring an architect to decide which drawer the cutlery goes in.
The candidate uses are narrow on purpose: classification, routing, evidence-strength checks, semantic duplicate detection where deterministic identity cannot resolve it, research screening, verification of completion claims, deciding whether deeper reasoning is warranted, and deciding what to retain in context. Each is a judgment; none is an action.
The limitation matters more than the capability. Type safety guarantees the output shape, not that the answer is correct. Jev cannot return a value outside your schema; it can absolutely return the wrong value inside it, with a confident probability attached. That is worse than a malformed response, which a parser catches.
We also concluded that raw thresholds such as p > 0.5 are too simplistic. Repeated runs on the same claim land on nearby but different probabilities, so a naked cutoff is not a production contract. What we specified instead: versioned, question-set-specific decision bands with an explicit uncertain state routing to human review, raw probabilities kept as evidence, runner-up margin considered for choice questions, and unregistered question sets failing closed. Bands get tuned from replay evidence and the asymmetric cost of a wrong decision versus a review.
What exists today is one reusable adapter through Vercel AI Gateway, with a timeout, kill switch and fail-open behavior, shadowing inbound classification and listing-identity scoring. Nothing in production reads its answer, and it cannot select a tenant, grant a permission, publish, contact a customer or move money. Confidence is not permission.
Jev is more interesting as a judgment primitive than as another general-purpose model. Typed output is not truth.
fast-jev-compaction — context is something you can filter
The standard answer to a long agent conversation is to summarize it, and summarizing is lossy in exactly the wrong way: the file path, the exact error, the constraint stated forty turns ago — the things that matter later — are what a summary smooths away.
Tran’s library inverts it: preserve the human conversation verbatim and rewrite nothing. Instead, score historical tool calls and their results for continued relevance, then keep, truncate or drop each. Recent messages are pinned, and if scoring fails it falls back to normal compaction.
Why this matters beyond the implementation: long-running AI workflows accumulate enormous volumes of stale operational context — a directory listing from an hour ago, a build log for a commit that no longer exists. The model pays for all of it in tokens, in latency and, most expensively, in attention.
The risks are real and we would not install it as-is. Dropping one old but still-binding constraint can be catastrophic, and the reduction ratio looks excellent while it happens — a compactor that saves ninety percent of the context and deletes the line naming the trusted target has not saved anything, it has made the next mistake cheaper to reach. Authority, permissions, the current target, outstanding errors and verification evidence should be pinned deterministically and never offered to the scorer. Upstream also uses a single probability threshold, the pattern we just argued against. Sending session context to an outside evaluator is a data-handling decision, and early-access model hooks should not quietly become production dependencies. So: bounded replay on sanitized transcripts, counting false drops, not bytes saved.
Context management may become as important as model selection.
StoryScope — sometimes the useful tool is an evaluator
A correction first, because we nearly made the mistake ourselves. StoryScope is research on detecting and characterizing AI-generated fiction, extracting an interpretable feature space of discourse-level narrative features — roughly 304 features across ten dimensions, such as character agency and chronological discontinuity. It is not a website copy tool, it does not measure conversion, and it makes no claim about business outcomes.
What interests us is one finding and one posture. The finding: AI-written narratives over-explain their themes and favor tidy, single-track plots, where human writing carries more ambiguity. Read that with marketing copy in mind. Over-explaining and excessive tidiness are the exact failure signature of AI-written business copy — the paragraph that states its thesis twice, the page where every sentence resolves and nothing is load-bearing.
The posture is the more valuable half: a rigorous, reproducible, interpretable evaluator of prose, built by people who cared about measuring the thing rather than generating more of it. We want to know whether that transfers into a diagnostic for questions we actually ask about a page — does this sound like normal human language, is the main idea understandable quickly, is the copy unnecessarily complicated, does the evidence support the claim attached to it, and which sections need revision. Findings separated from proposed rewrites, with exact references, and no automatic publishing. The honest risk: such an evaluator can sound authoritative while merely encoding one paper’s stylistic preferences. It gets one vote as a signal and none on business judgment.
Good AI tooling is not always another generator. Sometimes the useful tool is an evaluator.
App Test Cycles — verify side effects, not screens
Debernardini’s project runs a desktop application through a real user journey on a real screen, using a decision model to choose UI actions and validate postconditions, with a coding agent reading failure logs and fixing the bug before the test reruns.
The project is interesting, but the pattern is what we are taking: bounded journey, observable postcondition, evidence, repair, rerun. The important word is postcondition. A test that asserts a screen appeared has asserted that a screen appeared.
This matters for AI-built software because the three signals we most readily accept are weak. A model reporting done is a claim about its own belief. A passing build is a claim about compilation. A successful screen is a claim about rendering. None is evidence the workflow succeeded, and we have been burned by exactly this: a green pipeline and a confident report over a production surface that had not changed.
Take a real revenue path — lead, agreement, payment, webhook, database state, onboarding. Every screen along it can look perfect while the webhook silently failed and no customer record exists: the customer has been charged and the business does not know they exist. The only test that catches it asserts the real state changed across the whole path — the order row says paid, linked to the right relationship, onboarding fired exactly once, and replaying the event does not fire it twice. That postcondition must be read independently, from the API or the database, never inferred from the interface that was supposed to cause it.
AI testing needs to verify side effects, not just screens and model reports.
ChatGPT Work and GitHub events — stop being the transport layer
We run more than one AI system, and until recently a human was the wire between them: one model wrote code, and getting a second to review it meant somebody copying context between windows. That relay is exactly the sort of work that should not exist.
OpenAI now supports event-triggered GitHub tasks in ChatGPT Work. The interesting part is not another agent — we have enough. It is the removal of manual transport: Claude changes code, the GitHub event fires, ChatGPT wakes and reviews the current durable state. GitHub stays the shared bus and neither model’s conversation memory is ever the source of truth, which is what makes it safe.
The general principle: when a platform natively provides event plumbing that satisfies the requirement, custom webhook and orchestration infrastructure is cost without differentiation. Nobody buys from us because we wrote a good webhook receiver. Spend the engineering on the business logic above it — priority, work claiming, duplicate-execution prevention, recovery. The failure case is worth naming: if every bot event wakes another model, the automation quietly burns more than it saves. A preview bot comments, which wakes a reviewer, which posts, which is another event. Event filtering is the difference between leverage and an expensive loop.
Use native event plumbing where it exists. Spend custom engineering on the business logic above it.
Capy — is execution infrastructure actually our product?
Capy gives each task its own isolated cloud VM with a planning agent and an implementation agent, branch isolation per task, automated pull requests and a review step. We looked at it as an example rather than a purchase.
The useful insight was not switch to Capy. It was that branch, VM and executor infrastructure is increasingly something you buy rather than build. That reframes a question every company building AI products should ask and mostly does not: is execution infrastructure our differentiated product, or are we rebuilding commodity plumbing because we started before it was commodity?
For us the honest answer is that it is not our product. Our disposition is still to watch rather than migrate, and the reasoning is the transferable part. We already have executor infrastructure with a measured defect and a bounded repair in progress. Swapping platforms mid-diagnosis would confound it — we would lose the ability to tell whether the problem was fixed or merely relocated. Migrating away from a problem is not solving it, and it feels far more productive than it is. So we wrote down the trigger, not the opinion: reconsider when the executor is repaired and five to ten real revenue tasks still show a deficiency a hosted platform specifically removes.
Buy commodity execution infrastructure when it is not your product — but do not migrate away from an unfinished diagnosis.
Connectors and operator interfaces — authority stays server-side
The last pattern is not a tool but something we kept noticing across all of them. Models increasingly interact with systems through structured tools and connectors rather than by owning the underlying application state. Jev receives program state and returns a decision; the compactor scores tool results; the test loop reads postconditions from an API. In none does the model hold the state.
Four consequences follow, as design constraints rather than observations. AI clients are replaceable, so nothing important should be built such that swapping one is a rewrite; structured interfaces are therefore the durable asset. Authority remains server-side: permissions, target selection and protected actions are resolved by trusted code from trusted state, never by a model’s output, however confident. Deterministic systems calculate exact values — prices, totals, dates and identity are arithmetic, not judgment. What the model does well is interpret, explain and propose. That is not a demotion; it is the division of labor that lets you put a probabilistic component inside a system that has to be correct.
Models interpret and propose. Code computes and enforces. Authority never moves to the model.
The synthesis
Put the seven together and the shape is hard to miss. The interesting part of the current AI tooling wave has moved away from which model is smartest, toward everything around the model: context, evaluation, typed decisions, tool interfaces, execution environments, verification and event-driven workflows.
A modern AI system increasingly looks like a loop — context, model judgment, tool or action, evidence, evaluation — and around again. The model is one component: an important one, but no longer where most of the remaining engineering value sits, because the models are good and the scaffolding around them mostly is not.
There is a commercial reading too. If the differentiated work is in context, evaluation, verification and interfaces, then a company whose technical identity is a wrapper around a frontier model has a problem, and one that has built business truth, bounded capabilities, permissions and evidence into its own systems has an asset that survives the next model release. The discipline is unglamorous: identify the mechanism rather than the pitch, check whether something you own already does the job, give every item one disposition, bound the experiment, measure it on real work. Most weeks that produces fewer adoptions than the feed suggests. That is the correct outcome.
context → model judgment → tool / action → evidence → evaluation. The model is one component.