Comment by amluto
7 hours ago
I’m fascinated by this thing and by the way it’s interpreting Jev. It’s very cool, but is it actually a classifier?
IIUC they took an already-trained “frozen” LLM and trained a little model on top that takes both a question and the hidden states after processing the input data and produces answer “probabilities”. (In contrast, the original LLM would have been run in AR mode to generate multiple output tokens representing its answer.) But then they used it for a purpose that isn’t really classification.
IMO there is a rather large difference between “is this email spam” and “what character should I type in this agentic workload”. The former is classification: there is hopefully a ground truth (is the email spam?) and the model is trying to classify the email. You would score it with a proper scoring rule. The latter is a strategy: there usually isn’t a correct answer, now or in the future. The model is playing a game consisting of repeated rounds, and the only way to evaluate it is to see how well it plays. You can’t even usefully compare it to the optimal solution because you may not know the optimal solution and you don’t actually need the model to produce an optimal solution.
I do think this approach is really cool, and it does suggest that one might be able to use a modern LLM to process an input and then extract the model’s next agentic step in a very fast, non-AR manner, with results comparably good to the usual AR decoding. And I think it’s very interesting to decouple the tokenized input representation from the model output representation, both because prefill tends to be faster and cheaper than AR output and because it’s never seemed particularly sensible to me that a model should be constrained to generate outputs at the cadence of one run through the model per output token. (AFAIK the main reason that models work on the same input and output token space is that this is how the pretraining process works.)
I wonder how to fit “reasoning” into this framework. Maybe have the question be something like “do you need to think further and, if so, what is your first thinking token”. But maybe something more clever is possible.
> The latter is a strategy: there usually isn’t a correct answer, now or in the future. The model is playing a game consisting of repeated rounds, and the only way to evaluate it is to see how well it plays
This is self contradictory. You can tell a correct answer as you said, by looking at the score. In my $DAYJOB I am making hundreds of RL environments that produce a score for each intermediate state.
I'm increasingly of the opinion that stuff like Jev and this are overfitting and producing illusory but confident "probabilities" that are complete bullshit. You can't see the underlying reasoning... but it's incredibly tempting for people who want to place faith in them. Rather than even trying to understand the complex system at play, it's easy to give up on trying to find reason, and just accept the second- or third-tier outputs of massively complex things that, on the first tier, are not necessarily reliable sources of truth.
Let's give an example: Suppose you ran a Jev that tried to determine when another Jev was wrong about something ...baseball games, let's say. The second Jev would come up with a perfect list of when the first Jev was wrong and when not to believe it. So now you have a second-order system that you believe more than the first-order one.
Only after 100 baseball games, the second-order Jev is only as good as the first one, it just inverted a bunch of games that could have gone either way. So hell, you make a third-order Jev that analyzes the first two...and its results are AMAZING when you look at the historical record! Only, you know, that's what's called overfitting.
Honestly, [edit: Fuck, I just wrote "honestly". I've been brain damaged by you-know-who] the invention of the "noul" is a bit of a giveaway. Imagine what junk bond traders could've done with that in the '80s. Not-not-not-falsy is how we all like our stock picks served up, right?
Reading the raw statistical output of an LLM as if it were an oracular source of truth is literally idol worship and gambling in the purest sense.
Take it from a guy who lost $10k on baseball this season having Claude rewrite my original code to reverse and re-reverse underdogs and favorites; there is no fucking "probability" of anything coming out of an LLM, even if your source of truth isn't an LLM but an evolutionary algorithm you designed yourself. Which would've performed better before the LLM started interpreting ways to bucket it and make up bullshit probabilities around it.
Anyone going down the Jev path is deeply misguided, but will see the light once they realize they have re-invented the magic 8 ball. Or that executive decision maker cube from the 80s with 6 random answers.
Here's your Jev. Note the preponderance of nouls ;)
https://www.ebay.com/itm/146783842141
If you happen to gamble or just watch baseball, you'll see that reality is not so easily reduced to one number. On the most basic level, a single probability number leaves out volatility, without which it's utterly useless for predicting anything other than a sequence it's already trained and fitted for. But as a gambling addict and occasional patron of oracles and gurus, it's very easy and tempting to mistake a clear-cut "probably" for whatever you want to hear. Listening to an LLM's heartbeat for probabilities is just a gambler's fallacy taken out to the 19th degree.
Yes, Jev is calibrated "from factory" on a bunch of tasks, but we can be almost sure our own bespoke tasks are not covered. So the model does not really know how to produce calibrated confidence scores.
What it would need is a calibration dataset on which to align. There is no calibration in the abstract, only relative to a set of test examples. A model with an uncalibrated output probability can be recalibrated using conformal prediction. You run the model over your calibration examples, get the probabilities.
Assume the new example's answer is y, and calculate its nonconformity score, higher means a worse fit. Count how many calibration examples have a score at least as high as that. Add one to this count, then divide by the total number of calibration examples plus one.
I love this answer and your fondness for "assisted" betting. Do you blog?
Thank you for that. But I don't.
I can tell you though, I'm not alone... a completely degenerate cokehead at my local bar just showed me a baseball prediction app he had Claude build for him on his phone, which looked suspiciously similar to mine. (Mine might be hand coded and backed by a symbolic regression A-Life engine I've been working on since 2005, but so what? Claude has helped me flip most of the predictions for various reasons, to equal "success"). Crude, and yet his app somehow included wind direction forecasts for each inning of every game. This guy is smart as a punter but has probably never used a computer for anything in his life beyond downloading STD test results.
It's a brave new world.