Comment by hodgehog11
2 hours ago
An LLM is a zero-shot classifier with a large number of classes. All you need to do is establish what the output means and you can fine-tune an LLM final layer for this task if you like (and others have done). A student of mine did this as an exercise two years ago, and it was cool, but not publishable.
I agree with you on the "ease of use" business though. No one thought to make this sort of thing commercially available.
But there is no hot take here. Jev is not some new paradigm; engineering-wise, it is a trivial modification to the existing pipeline. That doesn't mean it isn't commercially viable.
No, all they had to do was come up with a quality post-training recipe, production inference stack that wouldn't fall over, GTM, documentation, schemas, etc. etc.
(also most signs point to this being LLaDA 2.0-adjacent so throw in solving some substantial mid-training)
I think it's 100% a hot take to call what they built trivial. Or at least it used to be.
There was a time when that kind of stuff was something between sour grapes and cluelessness about the gap between an idea and an actual commercial product deployed at scale, but now that's just weirdly normalized.
In fact, if anything I'm the weirdo for repeatedly taking issue with the way people are trivializing it ¯\_(ツ)_/¯
Someone else posted the jevbench site which compares jev to a bunch of other models. If you look only at the accuracy dimension:
https://benchmarkheaven.com/jev-models?w=100-0-0-0#jevc-weig...
Jev actually isn’t anywhere near the top. It even loses to open weight clones. This tells me that whatever their “calibration” dataset is, it doesn’t seem to be anything special.
Architecturally, it is trivial. That's something the community would have consensus on, so not a hot take.
I see your point, but Jev doesn't exist in a vacuum. When one (like me) says "trivial", they mean it relative to other attempts and developments in the field, all of which require everything you've mentioned at minimum. Commercialising any product, and doing it well, is hard. But the R&D factor here is substantially more straightforward than almost any other product in its category, because there is no architectural breakthrough here.