Comment by jacobgold
14 hours ago
First, congrats to the team on launching something genuinely interesting and new.
Seems like a more accurate title would be "Jev: Trading general purpose generation for fast typed inference" or something like that.
This is interesting, but the speed comparison seems misleading? A generative model that can output code in a Turing-complete language can do anything a computer can do.
Jev can only generate structured output, right? This is probably super useful for classification/routing/scoring, but it's nothing like the code generating models we're all using today for code and automation.
Also "can't hallucinate" seems wrong? Sure, it can't emit an invalid type, but it can still emit a completely wrong valid value. You can enforce structured output from an LLM too, with an appropriate harness, etc.
Assuming there's no funny business, the Doom demo is cool.
When they say "can't hallucinate" they mean they produce a confidence value for every result, so you could see for example it has 0.1 confidence, and you can disregard the result - that'd be different from hallucinating where it believes it's correct
if it puts a high confidence value on a wrong answer, thats still hallucinating, no?
llm hallucinations are high probability tokens that are incorrect vs the real world
No, I don't believe so. Hallucinations are not "high probability" in a real sense. They are an artifact of the random walk the inference algorithm takes, which causes it to latch on to and chase attractors in the noise. This random walk behavior is necessary for chat interfaces to be useful, but are less critical to typed output predictors. I'm guessing they found some optimization that is possible if you give up caring about chat.
Correct, they have not made a universal all-knowing omniscient oracle, which is what would be required for "can't hallucinate".
7 replies →
What we would want to see if a confidence value that is in line with the actual correctness. If the value is 0.9 for 1000 different answers, then approximately 900 of those answers should be correct.
Yes, there is no magic sauce here that makes stochastic output binary if that’s what people are looking for.
2 replies →
Technically speaking when you send the prefix “The capital of France is “ into an LLM it will also produce probabilities across its whole vocabulary.
The probability values don’t really represent confidence in modern LLMs though, especially after RLHF and RLVR.
System One says they use RLCD, Reinforcement Learning for Calibrated Decisions, which presumably has accurate probabilities as an explicit optimisation goal.
2 replies →
… which they could provide in their APIs but are vehemently opposed to because it makes distillation much easier, and faster.
1 reply →
that's right, but because these models are probabilistic, it's also possible to be confidently wrong (and all future models will be smarter still and still have that possibility)
I read "hallucinations" as "generates novel output with no grounding/source". i.e. "it just made something completely up".
I believe their "accuracy" metric (sonnet 5 level) is where "right/wrong" is measured.
Yeah but what stops it from producing confidently incorrect outputs...
Nothing, but imagine using LLMs for a classification task
People out there are so resigned to the models being unreliable that they are really doing things like hallucinating deliberately, and then matching the hallucinations to embeddings -
https://softwaredoug.com/blog/2026/08/10/hypothetical-classi...
You could do that or you could just... use a model that will never produce unreliable outputs in the first place.
4 replies →
What about the LLM calls though that are done midchain? In the Home Assistant video the multi-intent prompt gets split using what looks like a traditional llm model, which I'm assuming is vulnerable to classical hallucinations.
That's really funny when you consider that generative models also don't hallucinate if you check up on them on every token generated?
Has LLM become so synonymous with Generative Transformer that other high-parameter count models that interpret language need a different name?
For all we know this might be a non-language-generative transformer e.g. a transformer where the decoder produces confidence scores rather than language. Please provide more likely architectures if you know them, I'm genuinely curious.
> Assuming there's no funny business, the Doom demo is cool.
The Doom demo seems very funny business. They're not feeding it video, they're feeding it a text description of what's going on in the game. It's not reading pixel data.
I think LLMs would play a lot better with that input too but Jev does seem to have a huge speed advantage; I don't know if the other models could do that in real-time.
In a case like this it still seems more appropriate to encode that data in tabular form and use a tabular foundation model
Forgive my ignorance. Tabular foundation model?
I think the meaning of can't hallucinate in this model is that the type won't be hallucinated.
So if the generated schema is for a tool call for calculator, then the numbers will be valid numbers for sure (and not random words).
To me, it looks similar to BNF schema already introduced and implemented few years ago: generally speaking - it limits the next token that is allowed to be generated, probs are drawn from a subset tokens.
(tbh, I'm not sure why it didn't pick up as a more standard interface to LLMs, as it made a lot of sense back then, and now.)
Yeah, I thought about constrained generation as well. I've actually done something similar with local models before. And you can even get a "confidence" score by looking at the logits (something along the lines of logprob("YES") + logprob("Yes") + logprob("yes") - logprob("NO")...
There's also a cheeky "one of the models hallucinated a link" in the wiki jump example that most likely could have been avoided by properly using grammars. You can setup constrained gen so that only valid options (say from a list) can be outputted. Their own inference lib likely does that. So comparing to one that doesn't is a bit cheeky.
That being said, after a brief look at the site I could see this working. Especially if this can be ran locally, the speed and cost can enable some workflows where you have this as an "overseer" layer over say a cli agent. After each step you run through a list of "questions" ("is the task completed?" -> yes -> "does the edit touch files it shouldn't" / "does the edit follow our code writing policies") etc.
edit: extra points if the "question" rubric is also generated by a higher abstraction model. Say "/goal Build out auth" -> generate_rubrics(goal) -> "Is auth implemented on all endpoints" / "Has code touched anything else than auth" / "is this following the best practices" / ...
AFAICT it is the same interface as you describe, but the underlying inference algorithm is fundamentally different, hence the speed gains. There is an application I am currently working on right now where this typed output predictor is the performance bottleneck. I'd be very interested to see how this performs.
I don’t think it’s misleading if you compare on the use cases they suggested. It’s faster and cheaper (no idea if higher quality), so it’s immediately interesting for certain things.
And if you buy their RLCD claims, this might be even better than huge models that know a bunch of irrelevant things.
What was misleading was the original title:
"Jev: New frontier model 40-400x cheaper and 20-200x faster"
I'm not the gatekeeper of who gets to call themselves a frontier model, but I don't think most people would count Jev in that group. It sounds false.
If their specific claims hold up, then it would make more sense to say something like:
"Advanced the speed/cost frontier for structured decisions"
It is frontier in the sense it is exploring an unexplored domain. I do agree on questioning the comparatives though. Speed/cost is indeed relevant for problems that can be framed as structured decisions only. The question is, would defining a structured decision model be a structured decision model itself? This would significantly increase the application domain.
I dunno, I would consider Waymo and Tesla to have frontier models.
I think AlphaFold and related are also frontier models.
Being an LLM does not seem like the qualifier for frontier.
4 replies →
Large language models are not the only type of model.
How is this not a frontier model? It's bleeding edge in its own niche. It's not a frontier LLM; however, applicable to many of the things people use LLMs for.
3 replies →
[flagged]
I'm biased but I wouldn't call it misleading - generating text is super awesome and flexible, (we describe that in the blog post - and I personally use string models all the time) but it's true you pay a high tax for autoregressive generation
> Also "can't hallucinate" seems wrong? Sure, it can't emit an invalid type, but it can still emit a completely wrong valid value.
that is likely true of all ML! perhaps we could debate semantics, but I don't think it's fair to say a random forest "hallucinates" in the way LLMs do
His claim was that the title is misleading, not sure how it's relevant to that claim that you use "string models" (full LLMs).
The original title before it changed less than an hour ago was:
"Jev: New frontier model 40-400x cheaper and 20-200x faster"
I'm going to agree that was misleading.
And on the second point:
>>Also "can't hallucinate" seems wrong? Sure, it can't emit an invalid type, but it can still emit a completely wrong valid value.
>that is likely true of all ML! perhaps we could debate semantics, but I don't think it's fair to say a random forest "hallucinates" in the way LLMs do"
Also going to disagree here, and I don't think it's semantics.
Type safety is not factual correctness.
> Type safety is not factual correctness.
I very much agree with this and want to hone in on where do actually disagree. Would you say a linear classifier hallucinates?
11 replies →
Just to be sure that I understand, you're saying that your model "can't hallucinate" because it only outputs a single thing, right? In this way, an LLM can't hallucinate either if I prompt it to do a classification task with a discrete set of possible outputs, right? (Assuming I reject non-conforming output. Actually, maybe what you're saying is that your system can't output non-conforming output?)
Yeah that's precisely correct.
For e.g. classification tasks, even in 2026 people are doing things like hallucinating deliberately, and then matching the hallucinations to embeddings -
https://softwaredoug.com/blog/2026/08/10/hypothetical-classi...
With TypeSafe it just picks the class (actually probabilities across classes), reliably every single time.
3 replies →
From a quick look at this it looks like it could easily generate natural language text by following a structured representation like UMR (Uniform Meaning Representation) or the similar representation the Abstract-Wikipedia folks will be working on for generic encyclopedic text (which will be heavily informed by Universal Dependencies). These are basically linguistically principled and frame-based counterparts to a programming language AST, that can be then converted to natural language (in a broadly language-independent way, to the extent that semantics and pragmatics make that feasible) via some sort of NLG rendering.
(To be clear, this one raw model does not support outputing a full AST directly - it wants to output "choice" among fixed options, "score" on a sliding scale, or a true/false answer (all of these with confidence scores attached), so building the AST/structure would be a code-driven (or even perhaps outside LLM-driven in some more challenging cases) multi-step affair where the model would essentially be playing a "game" of building the structured output step by step and getting a revised partial state back. But one could expect this to lead to interesting results.)
> I'm biased but I wouldn't call it misleading
- @CompleteSkeptic
Very strange.
> Jev can only generate structured output, right? This is probably super useful for classification/routing/scoring,
My first thought was that it would be ideal for robotics? As in control of limbs, general planning, route finding, etc.
That's a great point! It quite looks like the System 1 model of Physical Intelligence
Um, Isn't SELF DRIVING the elephant in the room?
Only if you think that everyone cares about self-driving. Lots of niches require structured domains; self-driving is just one that has a lot of capital thrown at it.
I feel like the power of the approach presented here is that it gives a model a proper "language" to describe computations directly vs moving tape silliness.
I foresee this to be the path moving forward - giving AI models understanding of the computation directly(as well as compositional rules) This feels like a short path towards total software in many areas.
Agreed. It's a wildly dishonest presentation of their product from many perspectives, which is a shame because it might actually have some good use cases.
The comparison between LLM speed and Jev speed is misleading, because they're using autoregression to generate all of the type names, all of the schema, etc. A closer comparison would be if the LLM was purely outputting the raw numbers. Even then, comparisons to LLMs are pointless because you could train a transformer on the same sort of task that Jev is doing and get even better performance yet again, and a smaller model. I suspect this is some form of stripped down diffusion language model.
You really have to do a lot of hand holding here, and map out your problem space manually, and very carefully, to get any sort of accuracy. For example:
> Keep each Score to one dimension. If a description says “punctual and smart and experienced”, the question is measuring three things, and an input that is high on one and low on another can’t be placed. Confidence drops and the score means less. Split it into one Score per thing and combine them in code
If you don't perfectly represent the distributions of possible answers then you'll likely get garbage results. As far as probabilistic state machines are concerned, I'd say creating the distributions of possible answers, and their hierarchy, is the actual hard part.
One of their examples is:
- "state": "I have asked three times now. Can I please just talk to a real person?"
- "Is the customer asking for a human agent?"
Imagine the users request is: "I want your human agent to call me tomorrow at 5pm."
Human conversation is fuzzy, getting useful reliable results out of this is going to be a challenge. Of course, you could add follow up checks like: "Do they want that now, or later?" -> if later -> "Do they want that tomorrow, or the day after?" and so on... But now you're building an LLM out of if statements. I am skeptical of whether this model has much utility for fluid language interpretation - I suspect it'll only be useful for scenarios where you've tightly constrained the answer space but want to use fuzzy language to describe it. Like:
- Question to human: "Would you like a support agent RIGHT NOW?"
- Their response: Yes | Yeah | Mhmm | ye sure (any possible yes signal)
Model input: "Did they ask for a support agent?"
Still... a tiny LLM could accomplish this sort of thing without problem. And that doesn't stop someone from saying: "No, not right now. But tomorrow." - and the tomorrow would get missed. I think this is why people haven't really tried this approach much already.
Also their Doom demo is on structured state, not on images. Meaning, the enemies must be being served to the model as coordinates (or the exact angle of projectiles that hit the player), otherwise it'd have to scan every pixel of the 360 degrees to know whether an enemy is in front of the crosshair or not. You can see from the map below that it's also choosing travel checkpoints/destinations through walls. So they've severely cooked this to make it look far more capable than it is in practice, and any speed advantage that is offered here is not factoring in the shortcuts it is taking, the training on the map, and the fact that it can cheat because the structured state it is using is not bound by obstructions.
Here is their docs by the way: https://docs.typesafe.ai/ - so you can understand how it works.
[flagged]