Comment by nater5000
9 hours ago
>Classification models have been around for a while.
I'm still trying to catch-up on the Jev stuff, but my understanding is that it's basically just a more efficient LLM when all you want is the LLM to produce a classification.
There's more to it, of course, but it's not just "generic" classification ML because it accepts arbitrary inputs and can produce probabilities over arbitrary classes. Not saying this is the first time people have done this, but typically classification tasks are more static and limited.
In the same vein, it's also not just an LLM with structured outputs (which have been a thing for a while) specifically because that is a very inefficient way to approach classification using this kind of architecture. Jev models are much more performant because of how limited they are compared to a full LLM.
So when you want an LLM, but you only really need this kind of classification from the LLM, then Jev makes a ton of sense. This makes sense for me, since I've definitely used LLMs for this kind of classification work and, even then, it kind of felt like using a jackhammer to place some nails, etc.
Happy to be correct, though.
But an LLM provider could very easily add a "Jev mode" to any existing model, right? LLMs already produce a probability distribution over arbitrary classes. Just tell e.g. 5.6 Luna “here is the user's question, you must respond ONLY with the words 'foo', 'bar', or 'baz',” run a single forward pass of the model, and report the normalized probabilities of 'foo' 'bar' and 'baz' tokens before the first output.
With such an approach you could even retain full reasoning capability
My hunch is that you would need some post training. On top of that; I don’t think the llm itself can read inside the transformer state although I can see how that could be enabled. (I feel that would open up yet another class of exfiltration opportunities)
> Happy to be correct, though.
Not normally one to point out a typo but this one made me smile