Comment by Oras
21 hours ago
I played around with Jev last night and did it for classification tasks that I used Gemini 2.5 flash lite with.
It’s a bit faster and bit cheaper, but this is compared to LLM. The consistency was nice to see, BUT, as someone who trained NLP models prior to LLMs, it’s just BERT with more data. I can see why people would want ready made one shot classifier, and I can see the value of sending multiple classifier in one call, but I wouldn’t call it breakthrough. And I believe many labs will replicate it in no time and might have it as part of their harness.
I see it as a wake up call for the tech community to go back to basics for most tasks instead of relying solely on generic LLMs.
Anyone who has worked in ML for 10+ years would already know that the usage of LLMs for everything is lazy, wasteful and a high degree of marketing on it.
Anyone who has designed circuits will consider CPUs wasteful compared to ASICs. This new FPGA technology is just a less efficient ASIC.
That’s roughly what I’m hearing.
The fact that general purpose intelligent classifiers can be dynamically hacked together by an LLM in real time to allow them to build evolving labeled and understandable networks that perform substantially faster than the LLM, and can act as an intermediate sorting and organizing layer for caching context or handling simple tasks, and a complete layman like me can assemble a teachable layer of these in a few days from an inexpensive service…
That’s wild!
And then you can identify where an expert system needs a more specific ML technique for efficiency within this network that overlays the SOTA model. Or manually adjust the stored context in each secondary “neuron”. And paths forward can run programs or take actions at relative high speed.
And you can share these with others and improve them as a group.
You could insert this at the datacenters at scale with a local supervising expert to prune and encourage proper growth. You could identify specific gaps in capability that need more training, and patch over them temporarily.
Then you train those corrections back into the general purpose model, or you identify highly efficient subsystems for specific purposes.
And this is just one way to use it. High speed intelligent workflows can live in this. There’s a spot for a local LLM to learn on the fly.
Maybe I’m way off base, but for the non-experts Jev seems extremely valuable.
That's the thing. LLM's can be used as a real-time teaching layer and build systems that can operate at deterministic speeds and escape hatch into a LLM when confidence levels drop. This approach has helped us save more than $1 Million annually against the straight LLM classifiers at scale (and pass that on to the customers)
https://sureshsubasinghe.substack.com/p/how-to-cut-agent-llm...
https://sureshsubasinghe.substack.com/p/the-god-model-fallac...
Also all of the mobile/embedded/resource constrained environments. Like sure my phone can run an LLM but it’s going to be bad and drain my battery.
I don't think either of you are wrong. The parent's assertion is that we've known this for almost a decade. BERT was highly usable for classification and sentiment analysis a whopping 9 years ago, despite being less than 0.5B parameters large. Similar-scale models like FLAN-T5 showed that it could be improved without substantially scaling up.
Today, we're extremely spoiled by trillion parameter-scale models. Our conceptualization of vibe coding relies on wasteful tool-calling paradigms, the one-size-fits-all mentality of LLMs is part of the marketing blitz to make people buy more tokens. It's lazy on the part of frontier labs, but also wastes electricity, time and money.
6 replies →
Dude, this is gold!
FPGA's are definitely not new. They've been mainstream for 20 odd years+.
See https://www.eetimes.com/fpga-market-to-pass-2-7-billion-by-1...
I would rate using LLM for tasks more specific ML can handle as a lot like using one's smartphone to snap photos, listen to music, set alarms, and play video games in preference to carrying around a fun cam, ipod, watch, and switch 2 everywhere.
For those who need to dive really deep into each specific avenue and squeeze maximal quality out, the photographers will be packing DSLRs and intense gamers will wait til they get home to strap into a PS5 or a gaming rig or VR or whatever.
But "can get 90% of anyone's needs met in this field, and can do the same in dozens or hundreds of other fields simultaneously" will remain the killer solution for anyone with lots needs that each have bounded depth.
I prototyped an NLP pipeline using an LLM because setting up the whole NLP skeleton was way too much of a pain. It was able to adequately simulate each piece. The nice part was that I could attack the problem from above, at any point I could just have the full intelligence of the LLM at my disposal.
I have, LLMs are less fragile, that’s why I like them. The ability to generalize isn’t just about being general purpose, it’s super robust, and so assuming the budget is there (I agree they are inefficient) end up performing better on many classical tasks that have ood inputs. Before LLMs / foundation models we all struggled with generalization and at least in the work I was doing people were independently converging to using bigger more general models for tasks anyway as compute got cheaper. LLMs are just the most popular version of this.
> The ability to generalize isn’t just about being general purpose, it’s super robust
I work with LLMs daily. 5 of my specialized tasks are outperformed by a custom model than a general purpose frontier model. The performance of my custom models not only beat them but are orders of magnitude low in costs and thus are able to be used by more customers.
1 reply →
That's part of the irony here I guess. In specialized fields, think computer vision, there were lots of teams whose innovative state of the art model was essentially just a function of the limitless compute they could throw at the problem. Now there are just people with even bigger sticks.
There are lots of scenarios where specialized models still are the only option for real time, power efficiency, and so on. And transformers and other tech behind LLMs can equally produce better specialized models. But no sympathy for those who confused compute with innovation.
I wouldn’t say lazy, LLMs are fast to use and much more cost effective especially if you factor the cost and time of training (data preparation, data cleaning, … etc).
It’s hard to justify several months to business when there is something off-shelf ready to use and doesn’t require domain specialists to run.
People have been having this same debate in a very similar way on typed languages vs untyped interpreted languages. I think that, in a similar vein, if you look at the trend over time:
- the addition and standardization (with incomplete coverage) of the solution of adding typing to Python
- how much people are re-discovering the value of performance + typing (e.g. Rust)
then I'm going to take a small leap and extrapolate that the trend will be similar here.
The equivalent of the "one off script in python" will be the LLM, and the long term stable and maintainable solution will be something much more structured and focused like Jev.
In other words, the "Bitter Lesson" (the famous essay)?
I thought one core result that led to LLMs was the realization that a specialized model is not necessarily better at a task than a general one.
That goes all the way back to at least to Stein's Paradox in 1955, sadly too few people get educated about Statistics and keep thinking specialized models will necessarily be better. If you want to estimate the batting averages of 3 MLB baseball players from samples, you are better off building a model to predict all of their batting averages than computing the mean from a sample of each one separately.
https://en.wikipedia.org/wiki/Stein%27s_example
Agreed... that said, humans will happily do something wasteful for a very long time if it's easier than the alternative.
why would you waste your time messing around with a team of expensive ml engineers and data scientists that produce vastly inferior to a llm.
We ripped out custom homegrown ml models that were developed in last 10 yrs and put an llm in its place. Its the opposite of wasteful. Even local gemma models are vastly superior.
There’s a middle option. Once you figure that out, you’d soon understand my point today or tomorrow. I’ve been in this field for 21 years and I use LLMs everyday. I also know when to not use them.
4 replies →
Anyone who has worked in ML for 10+ years has heard of the Bitter Lesson, and doesn't want to be its next poster child.
This is why you shouldn't assume that my comment was a juxtaposition of LLMs in comparison with hand crafted feature ML models. The binary thinking is highly problematic IMO
The breakthrough is the product (maybe, haven’t tried it yet) just like the LLM breakthrough was ChatGPT. People were using the gpt completion api flying low for months if not years before, but it took that particular launch to reorient everyone.
There have been other "universal"/general classifiers like GLiNER, GLiFormer, etc based on BERTs (Laya itself is based on ModernBERT!), but I do think there's something underrated about slapping classification on a "big" model like I've seen post-Jev announcement, lots of Qwen stuff, but the most interesting to me so far is razorback16/openjev using DiffusionGemma. There's a level of generalization that lots and lots of parameters get you that you can't really get out of small models.
> using DiffusionGemma.
That's an interesting choice. One question I had when looking at the jev copy on their blog is if one "line" in their output looks / attends to other lines. I think not, since they say it's parallel and not autoregressive. In that regard, it would be interesting to play with diffusion, and see if you'd get better results by playing with types, locking some, and so on.
> That's an interesting choice. One question I had when looking at the jev copy on their blog is if one "line" in their output looks / attends to other lines. I think not, since they say it's parallel and not autoregressive.
I don't understand the connection between the lack of autoregression and options attending to each other.
Non autoregressive models can attend to all the inputs simultanously.
An autogregressive model can can attend to all the options in the context of each other by simply writing the options out twice. Autoregressive models actually requires this, since one of them will come later, and the earlier prefill inputs can't attend to the later ones.
I've come to the same conclusions as you.
> I see it as a wake up call for the tech community to go back to basics for most tasks instead of relying solely on generic LLMs.
I always say the cheapest LLM request is no request at all.
What's the cost (broadly speaking, not in your specific case) of doing the same work an LLM would have done without the LLM though?
In the Jev use case, LLMs are horribly uncalibrated. In general, they will not produce good probability estimates.
Their generality also comes with a latency/computation costs.
1 reply →
The data labeling objection baffles me. Even if you don’t need labels for training, how do you know your model is working if you’re not evaluating it?
My company specializes in statistical long document text classification, but nowadays we mainly work with audit trail requirements because we got tired of hearing complaints about our 5 example learning curve. Seems like the industry standard is telling an llm to label and telling an llm to eval, and crossing your fingers that it’s correct.
> it’s just BERT with more data
Let's take that as a given. Is BERT with more data not useful?
> I can see why people would want ready made one shot classifier, and I can see the value of sending multiple classifier in one call, but I wouldn’t call it breakthrough
Are those things that people want less useful because of what someone else calls it?
> I see it as a wake up call for the tech community to go back to basics for most tasks instead of relying solely on generic LLMs.
Maybe, or maybe to use Jev, which is useful?
Whether something is overmarketed or undermarketed, novel or derivative, it does not change its function.
I made it clear that it is useful and I can see many people using it including myself. My point is it’s not a breakthrough.
Fine tuning small models is not novel. The novelty is large model generalization without fine tuning, at small models cost/latency.
The OP acknowledged they needed to fine tune their model to the training data of the task vs. zero-shot Jev
If nothing else, it's a popularity breakthrough to have people excited about it.
Agreed, refreshing to hear others share this feeling.
Timing is important here, LLM's raised awareness of ML techniques but we need to remember that most don't have traditional ML educations, so there'll be some "rediscovery" like Jev
the huge benefit in real systems for Jev like solutions i see is the cybersecurity / prompt injection mitigation. since the output will always be well structured, there is no way prompt injection might make the system do something crazy.
probably a prompt injection can still affect the output though, in unforeseeable ways.
You can use structured outputs and validate them against a schema today. I do agree making it a hard constraint instead of best practice for developers closes a whole class of bugs.
> a bit cheaper
Gemini 2.5 Flash Lite is $500/Gt, Jev is $42/Gt. AKA an order of magnitude cheaper.
> BERT with more data
It is specifically not just that, in the same way that models which have been chat/task-optimized via RLHF (which made these models much more useful for a huge variety of tasks) are not just "the base transformer model with more data".
This, intuitively, feels like a "lower level, basal, reflex" layer for the LLM's intellection.-
>I believe many labs will replicate it in no time
I really doubt this actually. To me, Jev is a great example ofcounter positioning. When you consider just how hyper optimized the labs are around auto regressive LLMs, and just how much money they have already invested and are pre committed to investing in an entire stack for auto regressive transformers... then responding to Jev becomes nearly impossible actually. They would just be giving up too much.
Just think, everything from their current sources of revenue, the sales use cases they tout, the marketing on the websites, the messaging to customers, then technically to the APIs, their internal batching and scheduling algos, their GPU configs, the chips themselves. ALL OF IT is designed with generative text models in mind. Jev breaks all of it.
I think basically no chance of a response any time soon.
I don’t understand how you’ve reasoned your way here.
How could Jev have possibly built something out of reach of a frontier lab providing the same or 5x as much resourcing to one of their teams to achieve? Which they can do because Jev has only received $40M of funding recently, so a round that is approximately what OpenAI is spending per math problem they try cracking.
In addition to that, these frontier labs have got extremely good at generating synthetic data and running generalised training pipelines. I can only imagine how easy it would be for them to build this internally vs Jev building it from scratch.
And then the final thing: one of the best places you might apply Jev is within a harness, behind layers that customers increasingly have abstracted from them. Frontier labs have huge incentives to do this as it could make their offering much better and cheaper. And whoever gets this first wins another big attraction for users.
My take on this is Jev is either acquired almost immediately for the benefit of the next 1-3 months head start for whichever lab acquires them or we get a similar model offered from all labs in 3-6 months or sooner.
It would be amazing to have big BERTha with per-token pricing on GCP or AWS. There are many times I am reaching for a cheap classifier with the general behavior of an LLM.