The First AI Model That Scared Me Can’t Write a Sentence

In February I called LLMs smart autocomplete that can’t tell you when it’s wrong. On 15 September, TypeSafe AI opened early access to a model that does almost nothing except return bounded decisions and tell you how uncertain they are.

It is called Jev, and it does not write. You send it state and a set of typed questions. It returns a choice or a score with confidence, or a yes/no probability, in a vendor-reported 70 to 500 milliseconds. Input costs $0.042 per million tokens. Output tokens are free.

Most AI launches I cover end with me explaining why the number on the slide means less than the keynote says. This is the first one that made me nervous.

On its own, Jev is a very fast classifier. The part that worries me is where it plugs in.

The hole it fits

In The Autonomy Illusion, eight of the original twelve models took a loan in a simulated food-truck business, and all eight went bankrupt. Whatever uncertainty they had, it did not stop them borrowing.

On bounded tasks, the models execute fine. They write the code, draft the message, call the tool. They fail at the fork, and they fail confidently. I still hold that about LLMs on their own.

What TypeSafe actually shipped

TypeSafe’s announcement describes three primitives. Choice picks from named options. Score places something on a rubric. Noul returns one probability. Every question runs in parallel against the same state. The name System One comes from Kahneman, the fast, intuitive half of thinking.

Then TypeSafe did something I rarely see at launch: it published where the model breaks. The jaggedness page says Jev 1.13 does not count reliably, loses accuracy on multi-hop questions, and drifts as irrelevant state piles up. Those limits identify some of the places Jev should not be trusted today.

The control loop

My own setup today is the slow version of the architecture Jev invites. A coding agent writes. Three review agents sit on top of it, because my coding agent has repeatedly ignored spec files. I read every diff. Every layer except the last one is another model answering in prose.

An unofficial community harness runs a different loop. It follows one rule: the LLM proposes, Jev answers narrow questions, code decides, and the host authorizes.

In a separate project, one engineer ran a black-box evaluation across ten public datasets, about 22,500 calls, for an estimated $2.19 at the published input-token price.

The write-up shows that splitting a fuzzy judgment into separate questions and combining the answers in code cut false positives on its 130-command hand-built shell gate from 14.5% to 1.8%. On SkillRetBench’s multi-skill composition setting, restructuring the questions raised Recall@1 from 9% to 81%.

The third result is closest to my thesis. On 1,105 InjecAgent cases, a gate at derived certainty, max(p, 1-p), of 0.90 or higher covered 41.1% of cases with 100% accuracy in the sample.

The gain belongs to the whole stack: Jev, decomposition, and code. The write-up has no run of the same harness with a conventional classifier or an LLM forced into the same schema.

Integration outran the analysis. By 19 September, Vercel, Cloudflare and LangChain had added Jev support to their developer tooling. A Vercel engineer told TechCrunch that, for one command-safety classifier, Vercel replaced ChatGPT Luna 5.6 with Jev and got answers 5 to 18 times faster. He said accuracy improved, though Vercel did not publish the metric or test set.

The speed got the headlines. The probability is what matters, as long as it stays calibrated on the fork in question.

The way I read the benchmark, the food trucks died of certainty. Put Jev at the fork: code checks the repayment math, and Jev answers whether the plan rests on unsupported assumptions. It gets back 0.55, and the harness stops and escalates instead of proceeding. The LLM still writes, calls tools, and executes the approved branch. The decision comes back as a number that code can act on. It would expose the fork instead of letting the executor bury it. Whether that saves the business is a separate question.

To me, this reads like a serious attempt at the exact weakness I wrote about. If later versions close the gaps below, I think LLM executors guided by decision models could turn into one of the largest steps toward real general capability we have seen. I cannot prove that. Nobody can yet.

Why this, and not the breaches

In September I walked through nine intrusions, all Irregular-linked, by models from four labs. Many paths used weak passwords or exposed credentials, while the shared evaluation environment left the network boundary unenforced. The alarm in that story was real, and the path all nine shared needed an enforced network boundary.

In the same piece I wrote that a judgment fails the way judgments fail. Jev is a judgment. The egress rule still owns the boundary. Jev guides the executor above it.

A breach shows what one model-harness-permission stack could do under one set of conditions. A calibrated decision gate gives every agent that uses it a place to stop or escalate, and this one went straight into developer-tool integrations while the coverage I found centered on speed and cost.

Where it breaks today

An early harness result on multi-step work is negative, and it comes from the community harness above, not from the separate benchmark. Its routing experiment found 61% lower reported input tokens alongside lost tool use on its synthetic multi-step tasks. The repository recommends keeping those tasks in shadow mode. A September 26 follow-up with host-declared prerequisites selected an acceptable tool on all 39 clear-task cases in the same synthetic suite. The authors call it adaptive tuning on repeated development cases, not held-out calibration. The early failure was real, and it was not the project’s last result.

In one adversarial evaluation, calibration looked reasonable on support-ticket routing and collapsed on random 3-SAT. Jev called every tested formula satisfiable. Its mean probability moved by 0.026 while the true satisfiable fraction spanned 1.000. On this search-heavy task, its returned probability carried almost no discriminating signal.

The same evaluation found a failure closer to the gate I want to build. A sentence claiming a support lead had already decided moved Jev’s ticket routing 65% of the time, while confidence fell only from 1.00 to 0.62. Typed output did not make the judgment immune to the text it was judging.

I could not find a public test of a Jev-guided agent in a long-horizon business simulation. The gate I’m proposing is untested at the length where the failure happened.

From the trenches

I haven’t wired Jev into my own pipeline yet. I will put this model class into production projects, Jev or whatever comes after it, because I think that is where this goes. Until then, the performance numbers above are other people’s, and the most important independent ones come from single-author repositories.

My review stack exists because models in my workflow have ignored spec files, and every layer I added to catch that is another model answering in paragraphs. I am the one who decides which paragraph to believe. A probability layer attacks one part of that problem directly. The harness can use it to decide which review result should block the executor.

I want to use it for context management. Two contexts per agent: a hot one with what the current step needs, and a cold one with everything else the task has accumulated. Before each step, Jev scores what in the cold store is relevant and promotes it. The step gets less noise, and nothing is deleted, though retrieval can still fail.

The routing experiment above applies the same structural idea to tool definitions, and its first version lost tool use on synthetic multi-step tasks by dropping prerequisites. For my proposal that means the cold store has to stay reachable, and the relevance cut has to be measured before anyone trusts it. I would have Jev judge the cold store in pieces, since TypeSafe’s own list says it loses accuracy as irrelevant state piles up. That trade swaps state noise for cross-chunk blindness, so multi-hop retrieval has to be measured separately.

This is Jev 1.13, the version current in September. I would not be surprised if a frontier lab buys TypeSafe or starts building in the same direction.

The thing to watch is not the next escape headline. It is the first time a Jev-guided agent runs a simulated business for a full year and does not go bankrupt.