Comment by snitty
15 hours ago
This is an example of why I'm a little stumped about why people are excited about Jev-style decision models. For any application at scale it'd be way more accurate to have a dedicated classifier using the right architecture (neural net, bayesian, etc.) LLMs drive the cost of making those models down to near zero.
Just use the LLM to make a thing that's actually good at what you need it to do.
Jev's launch was the best AI era marketing campaign I've seen so far. It was executed and timed so perfectly that Jev became synonymous with decision models overnight.
I think it reveals how many people were only familiar with LLMs as the generic AI tool for everything. It only took one ambitious marketing campaign for someone to come along and present a simpler technology as a new breakthrough to audiences that hadn't considered anything else.
The largest LLMs are really amazing for how much they can accomplish on unseen tasks. That is undeniably remarkable. The mistake that gets made everywhere is thinking that they're the best tool for every task, because LLMs are the only AI tool in most developer's AI toolbelt right now.
>many people were only familiar with LLMs as the generic AI tool for everything
That’s exactly how the biggest vendors have been marketing them, it’s no wonder non-experts think of them that way. I’m not an expert myself but have a passing familiarity with theory as a SWE and grad student who did some research for machine translation and currently am in a computing-adjacent field. I use LLMs heavily for work and have never come across any coder-oriented tutorial, guide, etc. encouraging use of LLMs as decision models specifically, even if they are being used that way “beneath the hood.” Jev’s announcement seemed sort of new functionality to me but even in the first post about it on HN (where I heard about it), iirc people were pointing out that LLMs do the same thing, at least to some extent.
I think you have a very narrow and specific experience with people claiming to be "devs".
I don't know of any devs in real life who would loudly and publicly get excited about anything unless it was their own completed project to show off.
The tools are a footnote. It's precisely because it's so hard to get a project that far along that they would be familiar with many more tools than they would ever speak about. I suppose some people have a low bar. I wouldn't really call them "devs", especially now in this "AI era".
> I don't know of any devs in real life
I think you're coming at this from a very narrow and specific experience with a few people you know.
Jev really went from the announcement to being discussed all over overnight. We're discussing how popular things are in general, not among your narrow and specific group of people you work with.
2 replies →
The appeal is that you can incorporate more context in a decision problem without knowing in advance what that context might look like.
Take the problem of determining if two people in a genealogy database represent the same person. It's a decision problem. There will be some data in structured formats you can count on, but which may be missing, or may even be wrong. There will be lots of fuzzy data (born in Newtown yes, but which Newtown?) and unstructured data (notes, etc.)
Now, do you want to try to assemble a dataset to train your own classifier on that, or do you want to throw a general purpose LLM-rooted decision model on it?
There are a lot of people working in software that have limited knowledge of information theory or machine learning.
Unfortunately LLMs are magic to them.
When you lack a first principles understanding of how something really works, you will bolt on hacks and duct tape on the wrong approach.
Hardly a surprise when the curriculum is vastly different between institutions with cohort going through an accelerated boot camp and others being told low level computing is not useful. There are resources for people that are curious and want to broaden their knowledge, but it has to come from the individual.
it's also the unfortunate practice of trying to use the new shiny thing everywhere and anywhere to signal that you're keeping with the times. It's funny how i've seen newer devs brush off things as 'traditional ml'
Which is nuts because one thing LLMs are actually pretty good up is generating "traditional ml" harnesses. You just have to describe the problem you are trying to solve and ask the LLM for algorithm suggestions.
It's also nuts because "traditional ml" generally requires a fraction of the resources of an LLM. If you need to auto label, tag, categorize, it's never been easier to setup that sort of system in any project.
I think there are a lot of practical applications where something that works few-shot and doesn't require training is quite appealing if the cost of inference is low. That's the sweet spot for Jev and similar models. If you have time and an eval set to train it, clearly a "real" special-purpose classifier should have much better performance (and for most applications cheaper cost/faster inference), but Jev and friends are way better than using a full-fat transformer LLM for a classification task.
As an example, I've been using winnow running locally as a decision model in openchamber to select between llms for doing various tasks. I don't have an eval set of different coding and other tasks that I could use to train a proper classifier. This I just download off huggingface and it works out of the box and has cut my token spend to a fraction of what it was. So it's good enough to be beneficial.
> Just use the LLM to make a thing that's actually good at what you need it to do.
This is the end state of most imagined use cases for “agentic” software IMO. Use the AI to build software, with a classifier or LLM as the expansion joint in awkward spots.
The frustrating part about an LLM is at a fundamental level it's designed to do the wrong thing, it's designed to provide an answer for things it can't do, e.g. one argument I've been in recently at work is someone using it to summarise numerical data. They've a raw data spreadsheet they're putting into ChatGPT and asking it to produce a report in a particular style. So the LLM is doing its job, predicting the tokens to emit, and not actually doing mathematical analysis of the underlying data.
I had a team proudly show me how they were feeding their dashboard data into an LLM to predict growth needs, instead of doing basic linear regression or any number of pre-existing basic statistical analysis. The part that depressed me the most on that was it was a bunch of engineers that did it, the people you'd most hope would understand what an LLM is/does.
For numerical work, there’s no reason to assume the computation is happening in the LLM. All frontier models could handily write a Python script to do linear regression then report the result, so the quality of the responses here depends entirely on the prompt + harness.
6 replies →
These days I wouldn't be surprised if the models are smart enough to get the analysis correct.
However sometimes I think that AI use shows a revealed preference around things. In this case, the actual data just ... maybe never mattered to them at all? People wanted a shiny chart and they got a shiny chart.
Either chatgpt or claude are building scripts to do those and has been for a few months at least now.
This is exactly what we’ve been doing.
For example, a client want a classifier to decide to make a go/no go decision on roofing jobs based on weather. The end result doesn’t use LLMs at all, but does use a lot of other techniques including human feedback when it makes the wrong call.
(Now the work crew gets the decision at 5 AM instead of 5:30 and the guy making the decision doesn’t have to wake up at all.)
(Presumably you mean the guy affected by the decision?)
2 replies →
I do agree with you, but I do think you're also making the classic "curse of knowledge" mistake.
You have to know the right words to ask the LLM to make and train a classifier. You and I already know the words. But they are not actually that trivial to learn for an average joe, and people are lazy.
The tooling around LLMs is still nascent and, in my opinion, quite poor. You may be able to shotgun easily-falsifiable mathematical proofs with raw compute, but get into the messy world of real data, uncertainty quantification, and a host of other things that go into building real world models, and I think that an LLM plus tooling dedicated to the task plus a human in the loop will open up capabilities not available to an LLM in a chat or development centric software environment.
I don't believe this is true with current frontier coding agents. Or at least, it's becoming increasingly less true, such that I think it will effectively be nullified in ~6 months.
I'm gonna cite myself here:
«We came full circle on the problem that also exists with Google Search, and why so many people don't get good results out of it.» [0] It is not that access to all information was hidden to anyone before, it was harder to find it, sure. Now it can be found easily and aggregated, however, it is not as simple as saying "Do X" and believe that X is correct. Recall the infamous Swing & Product Management meme? [1] The problem is the absence of information you provide, the LLM will have to make a guess of what you mean, much like the human. Likelihoods of this guess being correct depends on the domain and background. So yes, it'll likely build whatever you ask it (I do think we're there already), but if that matches what you intended to actually get is vastly different. You'll be settling with "Good enough" though and declare it working.
Others already pointed out that 6 months seem to be the constant of distance, irrespective of the date.
[0] - https://www.productftw.com/productftw-2-the-tire-swing-carto...
I'm done waiting 6 months. Just let me know the night before when I don't have to come in to work tomorrow.
5 replies →
[dead]
LLMs are the perfect tool for "finding the right words". They are literally translation machines.
Just ask the LLM for these words, obviously !
It's funny you mention this, because I've been thinking lately that all the focus on code quality benchmarks for LLMs seems to miss something. In my experience, LLMs aren't actually all that good at prompting other LLMs, and that's exactly the skill that matters for proper subagent usage.
The corpus of code available to train LLMs on is rich in both volume and quality. The same can't be said for LLM prompts. Prompts only came into existence alongside LLMs themselves, so most of the prompts out there are actually LLM generated, and training on them causes every model to overfit to the same patterns.
LLMs can give clear and concise directives to other LLMs, but only for basic tasks. Beyond that, they frequently overstuff their prompts with too much prose and too much direction, and worse, they often fill them with subtle contradictions.
Of course, there's a spectrum here. Some models turn out to be better at this than others, and I suspect this is what differentiates Fable-type models from the rest, and why vendors have started keeping their models' reasoning confidential.
2 replies →
That's what I've been doing in areas I lack knowledge tbh. Instead of a half assed attempt at describing what I need, I ask an llm to take my base idea and write it up using industry standard jargon as if I were commissioning a team to do the work. Has been working pretty well so far!
Okay so I just asked it "What question should I be asking you but I don't because I'm not smart enough to think of it?" Response was kind of interesting I have to say.
2 replies →
So what? The average joe shouldn't be trying to diagnose people with cancer.
Healthcare is more than having medical knowledge.
Ten years ago, in France, two doctors (one in a professional setting and the other during a private consultation) asked me about a wound I had on my shoulder; obviously I couldn't identify what it was, and they offered no suggestions, so I supposed it was not something important.
It took a visit to a cardiologist—for an entirely different reason—for him to suggest that I rapidly consult a dermatologist.
It was a carcinoma, and surgery removed it. If ChatGPT had existed back then, this carcinoma would have been removed several years earlier.
13 replies →
Prompted LLM classifiers are good for when training data is expensive and/or you don’t expect a high inference volume. It’s basically choosing lower model development costs in exchange for higher inference costs, which can be good for TCO, time to market, or both. Heck, I’ve worked on some applications that weren’t even feasible before LLM-prompted classification because of the cost to acquire enough data to train a supervised model.
Cost/benefit tips in favor of supervised machine learning when training data is cheap and/or plentiful, or when you expect to be able to amortize model development costs across a whole lot of inference.
It seems to me that hype cycle of jev made it harder to understand rather than easier.
Universal classifier just seems nonsensical. It seems clear that it can't have accurate probability distributions on any possible question you might ask it. Moreover, giving only "maybe" answers to everything adds a layer of equivocation that makes it harder to catch it in a clear mistake.
That said, I've encountered plenty of fuzzy judgment questions now (ie is this case more like example X or example Y?) that reasoning models struggle with. I could see how, in lieu of a dedicated classifier, a model like jev, ostensibly trained to guesstimate the relative likelihood of natural language hypotheses, may do well.
But it still feels wrong to call it a classifier in the traditional sense, let alone a universal classifier.
This is like saying we don’t need Dropbox because we have FTP servers.
People vastly underrate the patience of people to learn and setup a classifier. Jev is one API call. Training a good classifier is hard.
I don't know, a lot of the things we used to take for granted as "hard" just aren't anymore. Writing a parser for a compiler? It used to be an endeavor that took several days, now an LLM will do it in 20 mins.
Same for annoying training jobs, with sufficient data you can probably let an LLM loose on it in autotrain mode and it will come back with a decent classifier.
> Writing a parser for a compiler? It used to be an endeavor that took several days, now an LLM will do it in 20 mins.
Heh this used to be an interview question I gave. You’d be surprised how far people would get in 30 minutes.
It’s not the software development that’s hard. It’s the data science and domain knowledge to build a good training set and a meaningful objective.
Training classifiers is shockingly easy nowadays with LLMs. This is probably the biggest single piece of bespoke software I ship that I didn’t used to.
We have in the works a classifier for if a prospective customer will be long term profitable or not. This changes everything.
> We have in the works a classifier for if a prospective customer will be long term profitable or not
I'm curious, why do you use a classifier for long-term profitability? Wouldn't predicting expected lifetime profit with a forecasting model be more meaningful than just predicting whether a customer will be profitable or not?
1 reply →
We don't need Dropbox, it's a heap of shit.
With enough marketing you can sell anything. Also a decent way to make a product is to do the hard part for someone else.
I've been training classifiers for ten years, it used to be hard. Now it's hard to do really well but it's trivial to do well enough.
> decent way to make a product is to do the hard part for someone else.
100% this is a very good insight
> Just use the LLM to make a thing that's actually good at what you need it to do.
Thinking about this every time someone shares their setup and there's LLMs doing simple periodical things directly instead of having the LLM write a deterministic Python script that will never try to hack your sandbox lol
What we've been trying Jev for is interpreting things like short product descriptions provided by customers, and determine which of the natural language categories fits best.
The list of categories are provided by the government and can change several times a year, and the product descriptions are almost random (think the various titles of AliExpress listings of essentially the same product).
What kind of dedicated classifier would be better? Genuinely curious.
People like me who are excited about Jev-style models aren't going to use them to diagnose cancer. We are going to use them to make fairly straightforward decisions very quickly (300 ms or less) and very cheaply ($0.042 per M input tokens - output tokens aren't billed). Even the lower end frontier models will take much longer and be much more expensive, so these decision models are perfect for certain things.
But as the GP said, you’d likely be better off if you used an LLM to help you train a custom model to do what you’re trying to do.
If you really evaluate the best options, it’s very likely that a Jev-style decision model won’t give the best results.
If you know the exact domain you're working in, then a trained model could potentially do better, but its not a guarantee. For example, I fine-tuned Muse Glimmer on a specific domain and then tested it against Mercury Decide - which I've found does better than Jev - and they were even (and no fine-tuning was done on Mercury). But Mercury Decide was was faster and cheaper. The P95 for latency was about 5.6x lower. Decision models are very, very fast.
Training a model requires labels training data. Getting up and running with an LLM just requires a prompt.
I agree that a good model will outperform an LLM, but the LLM is so much faster/easier to build. That’s the usual tradeoff.
I just ran an experiment, I took the banking77 dataset, took the labels out, and got an LLM (Opus 5) to label them, then trained a logistic regression classifier (Jeffy http://github.com/nicobrenner/jeffy) on the resulting set
Got the same accuracy and f1 from the Opus-label dataset than the original one (I realize that Opus might have banking77 in its training data, so might need to account for that somehow)
But the point is: it's way cheaper to run the LLM model over your data to label once, and then use a cheap classifier, than using the LLM directly for classification
That’s true. It could help with the cold start problem for training models and maybe replace heuristics/rules in some cases.
> For any application at scale it'd be way more accurate to have a dedicated classifier using the right architecture
Agreed, and that’s the premise behind Jeffy, open source, bundles pretrained task-specific models that run (and train) locally on CPU: https://jeffyclassify.com/
At the same time, I’ve been working with some people implementing it, and there’s definitely value in having a more general model like Jev, or an LLM, for initial exploration and data labeling, to then use that data and train the task-specific model. It’s a whole workflow
Also, the market/pain insight of Typesafe is super interesting. They realized that a big chunk of the market was using LLMs as classifiers, and they built a solution around that (very much to your point, a task-specific model, rather than a more general LLM), and it worked! (Also they have the ability to generate a lot of attention, given the founder being ex-OpenAI and having raised quite a bit of money too), so it’s not just a technical/engineering thing, it’s a business/market/need/discoverability issue as well
A classifier requires a ton of labeled data to be processed for it to be good. Jev and likes are zero shot classifiers where none of the data is required. Imo both are useful for different tasks. I am not going to use jev based models for compliance or any tasks where being wrong can bankrupt my business. For something trivial like sentiment analysis it’s a god sent.
I've found that while this technically true, if you're building a classifier of any importance, you're going to need to monitor it and grade the efficacy of any changes, which will require labeled data anyways.
This kind of approach works for one-shot "let's just see if it works" type stuff, but anything that I would stand behind needs at least some level of labeled data and curated dataset.
Can’t you use existing open weights sentiment analysis models that are far more lightweight though?
There's a significant engineering cost to having a custom classifier.
An off-the-shelf API has way less friction, if the results are good enough why bother?
For this particular case, it's probably worth it because a dermatologist visit is $200. If I were the NHS or Kaiser, I'd partner with other integrated insurance and service providers to collect as much training data as possible and fund classifier development.
This is the plan!
There's a great bit of shortform video content floating around that shows a stammering younger guy in a welding mask:
That's Jev, and ~all zero-shot classification-by-LLM. It's universally OK. And that's great, for the tranche of problems where OK is good enough. But anywhere OK is not good enough, one would be advised to use or build something somewhat more robust.
Once you have a hammer everything is a nail.
True for modern hammers while older had weaker decision models limiting them to finger nails.
Even the hammer?
Mc Hammer?
Classifiers can require a lot of examples to train correctly. You can take a decision model's outputs and apply platt scaling to arrive at a good decision boundary a lot faster.
Eventually, yes, it makes sense to move to a pure classifier model as it's probably a lot cheaper. But then you need to learn how to serve your own classifier model correctly. Compute costs are being driven up by demand to serve models and unless you're careful it's easy to use your GPUs less effectively than a company specializing in serving a model.
Use LLMs to implement, process data, test and iterate statistical and NN models.
The LLM is like the swiss army knife in these situations. It can KINDA do it all, but not very well in a lot of cases. Better to have specialized tools in most cases. And you're right, the LLM is pretty good for figuring out or building out the right tool for the job.
I think there is a parallel between engineering systems (specialized) and commodity cloud hardware (general purpose).
While no doubt, you can get better performance using a specialized hardware … there becomes a breaking point when the commodity / general purpose is good enough while allowing you lots more flexibility as well as typically being cheaper that is become hard to justify the specialized solution.
>LLMs drive the cost of making those models down to near zero.
The last place I worked and my current job have a tremendously bureaucratic process for standing up new internally trained models. You could easily spend ten times as much mental energy and effort doing the paperwork and revisiting it annually as you do in the care and feeding of the model itself.
If a canned product meets the requirement, there's a lot to be said for using it. Even if it costs a bit more per decision, the all-in costs are WAY lower.
It is often better to use a pretrained model. But the choice here is using a frontier LLM to classify or a purpose built classification algorithm.
I know which I’d expect to perform better.
> The last place I worked and my current job have a tremendously bureaucratic process for standing up new internally trained models
Have you considered putting an LLM on the task of completing the bureaucratic gauntlet
Looks like you might be getting downvoted but yes!
> Just use the LLM to make a thing that's actually good at what you need it to do.
I don’t have a deep understanding of this topic, but wouldn’t you also need a good amount of relevant data to train on?
Not if you take advantage of open-weight models and make an adaptor or distill its knowledge into the thing you want.
> why people are excited about Jev-style decision models.
I think its a combination of being new to classification models and not using in anger.
LLM Decision models are still relatively new and suffer from a number of issues. Order of choices impacts probability, it starts to break down over 20 classifications and even how you word things impacts probability.
> and not using in anger
Is this a typo? are you saying that this is so new people haven't struggled with Jev enough yet? very interesting
> it starts to break down over 20 classifications
Yes, I recently ran a benchmark on exactly that: https://jeffyclassify.com/blog/when-llms-fail-at-classificat...
At around 4 classes already LLM-based classifiers (Jev/Laya/GLiNER et al) start falling behind
> Is this a typo?
My understanding of the statement is trying to use it in production for the first time. Once you start to actually use decision models beyond something simple you start to see the limitations.
> For any application at scale it'd be way more accurate to have a dedicated classifier using the right architecture
There’s lots of applications where it’s not being deployed at scale. Also lots of cases where someone who knows they need something classified does not know what a classifier is.
I have been incredibly impressed by LLMs for coding and data analysis and was expecting the LLMs to perform better so I was surprised by the results especially since this is an old model and nowhere near state of the art for skin cancer diagnosis.
A transformer trained only on cancer images could probably do better but the old model is trained on a well curated data set to do one thing. A decision model can decide on a much broader universe of things, only a sliver of which is cancer (and probably there's a smaller cancer dataset too)
It's because you can "program" it in casual English. No code, no training runs.
For some types of applications, Jev is also useful as a stepping stone, you use it for the cases it's most confident so you already get gains from using it, and then collect all the usage to train your specialized model.
You can train gradient boosted trees on jev style features, it is a quick “good enough” approach for a lot of stuff
They make very good classifiers and are easy to fine tune for specific use cases.
Don’t those decision models mainly route to other models? My understanding is that they wouldn’t handle a request but forward it along with some kind of semantic capability.
> Just use the LLM to make a thing that's actually good at what you need it to do.
Yes, this.
it's precisely because you can tune them to do that off a strong base
and because they start off generalized, you can train them against multiple of your workloads (they'll start with some generalized performance/world knowledge about them)
Which allows you to embed proprietary processes/data (world knowledge only you may have) across multiple domains into a model cheaply
Yes, of course, with time, attention, higher maintenance costs, and slower time to respond to requirements changes you can build a specialized system that performs better than an off the shelf general purpose solution. But you cost money. And you can quit and go work somewhere else, stranding the project. And the data might be garbage you can’t train on without substantial effort. And training takes time, which has opportunity costs that are often substantial. And the requirements may change tomorrow with no notice, invalidating all of your work.
I don’t want to pay someone to maintain a first rate classifier when I can buy a pretty good classifier as a service. I don’t want to pay someone to maintain and build server infrastructure when I can buy that as a service.
I feel like I’m being gaslit on HN when a solution seems obviously valuable to me because it eliminates or modifies constraints, and people dismiss that as if managing constraints wasn’t most of high level engineering work.
I tuned out of HN for a few years, came back in the middle of the AI renaissance, and I am constantly raising my eyebrows and hearing this Upton Sinclair quote in my head:
“It is difficult to get a man to understand something when his salary depends on his not understanding it.”
I think the contention is CNN vs Transformer
> For any application at scale it'd be way more accurate to have a dedicated classifier
"at scale" or the above use case of "specialized domain"
Jev being cheaper than an a full LLM call creates places where other tools might work better later - when the volume or the available data warrants it.
System optimal, code optimal and cost optimal aren't always aligned. Pick the right tool for the job - and dont get married to it. On occasion worse is better, and moreso when your testing or it is a one off/disposable.
I mean, it's the pareto principle, right? 80% of the applications require 20% of the intelligence. If I can drop in something without needing to create high quality training data that's good enough, that's great! Skin cancer probably isn't one of those applications.
The only real usecase I can see for jev style like models are processing long lists of items. i.e server logs or financial transactions or sensor data in robotics.
Why?
Internet discovers If statements
subnanosecond response times
model fits on CPU
[flagged]
[dead]