Comment by wuhhh

1 day ago

I don't understand how this is different from oai "structured output" (and whatever the similar paradigm was on Sonnet ~3.7 back then) which everyone moved on from. On their gh they say:

"Jev is TypeSafe's closed service for runtime-defined semantic decisions. This project reproduces that interface pattern with open models; it does not reproduce Jev's undisclosed model or training"

As someone else pointed out it isn't actually Jev... can someone enlighten me

25 comments

wuhhh

Reply
Topfi  1 day ago

Jev is, as far as I understand, essentially very optimised for zero shot classification [0]. Something like BERT could be and has been tuned to provide similar "decision making" at a similar latency and cost advantage quite some time back. Advantage over full on LLMs is mainly the efficiency and of something like Jev over e.g. the encoder/decoder based classifier I had in front of an LLM to route to different prompts depending on the users likely needs, that Jev does perform at a more consistent level, allegedly roughly akin to GPT-5.6 Terra, but at the lower cost and latency. Currently testing that, but seems promising, if Jev classifies at or above Terra level, I see no reason not to leverage it.

Can add that I tried using a heavily pruned mt0 based model for structured classification along with structured output for local tagging and simple renaming suggestions. While it does work, the balance is hard to get right for the machine I was targeting as a minimum spec (Macbook Neo), so that's on ice. Focusing on one of the tasks easily goes below 100mb with solid latency across all EU Latin script languages, but the second you add a few, it's simply not in the quality budget, so while LLMs can do anything Jev and similarly focused models can, it comes at a literal cost. Could maybe accomplish the goal with multiple models (BERT+mt0+...), but that get messy.

In general just happy to see a bit of the millions flooding into the industry being used to improve on less flashy but immensely useful solutions. It's amazing that you can technically use LLMs for most tasks, but not every org has a near infinite budget and there is still a lot to gain from applying more recent learnings to old solutions along with just updating their training data to the current year. Also makes business sense, competition on frontier or mid-tier LLMs is vicious, focusing on an underserved niche with clear application is clever.

[0] https://huggingface.co/tasks/zero-shot-classification

orbital-decay  1 day ago

It's a non-instruction-tuned classifier model trained on a confidence-aware RL variety that generates its own schema and follows it, with a confidence score output. Think BERT on crack, smart enough to be used as a decision maker (conceptually). They call it "not an LLM" because it's non-generative but of course it's a language model in the same way all non-instruction-tuned classifiers are.

  • mtkd  1 day ago

    I was a bit skeptical when read the initial pr on it, yesterday ran a test involving ~250M tokens, something we measure went from ~60% to >80% success (with almost no tuning) and at less than 50% cost the low-end LLM was running at, looking at it more seriously now ... the servers are US-only currently I understand and ZDR is by request

  • 3abiton  6 hours ago

    This is a great evolution in the right direction compared to LLM + pydantic and temperature 0.

  • ozgung  1 day ago

    Isn't that the same transformer at the end of the day? It must be faster only because it generates a single token output, just one evaluation of the model. It takes the same input context and has the same O(n^2) attention blocks. It probably takes options as appended to the input and returns a probability over them instead of the whole dictionary. It's post-trained to do that specific job. If so what's the big deal?

    • orbital-decay  1 day ago

      They say it's "parallelized". Whatever that means in reality, their demos are pretty good, their prices are extremely low compared to alternatives, and it responds in ~100ms which is pretty fast for what they do. Whether it holds for longer inputs, edge cases, etc. remains to be seen, but I can imagine the use cases for that, for example you can use it directly in the sampling layer of a normal generative model, or just as a generic decision maker/controller. They can (and will, in their words) do this for images too. I don't know if it's a big deal, but it's kind of a fresh perspective.

      2 replies →

messh  20 hours ago

In Jev you pass options in the input and its output just gives some probability for each. Oai structured output just follows a schema. The exact output is still generated and there is no probability

  • brokensegue  14 hours ago

    you can ask structured output for probabilities...not that they necessarily mean anything.

    • cheesecakegood  11 hours ago

      In theory Jev “calibrates” the probabilities, meaning a probability of 20% is optimized to happen near 20% of the time, which as you point out traditional models with schemas do NOT optimize for at all

mritchie712  1 day ago

in short: it's faster, cheaper, smart structured output.

each "question" is answered in parallel instead of a sequential (like an LLM). so if you have an input like:

    {"is_it_hotdog": noul, "is_it_apple", noul}

it answers is_it_hotdog and is_it_apple in parallel and gives a probability.

jLaForest  1 day ago

I'm in the middle of moving my app to openAI structured output.

Could you please explain what you mean by "which everyone moved on from"?

  • slickytail  1 day ago

    Structured decoding limits next-token probabilities to ensure valid JSON. The main issue is that if the model puts a substantial probability on an invalid token, then it was already confused, and in that case, you don't actually want whatever the next-most-likely valid token is: even if it's syntactically valid, it's likely semantically erroneous.