To be honest, I believe I get the point the article is trying to make, and to an extent I agree, but I also think the point is not really made very well.
The core of the argument as I understood it is that LLMs aren't just using existing data is training but also new ones. That's fine and good, and you can't simply assume an LLM is simply mashing together all it's data to give you an average of all that got fed into it - but at least I would still call it a "next token predictor"
It's not using just training data, but what it's doing is predicting the next token to get to the solution. As far as my amateur knowledge goes, LLMs still roughly go token by token, deciding which one fits best given the context.
It's just not predicting based on it's training data, but predicting based on RLVR & more, trying to get to the optimal solution ( as much as the solutions CAN be optimal)
And I honestly think keeping this very much in mind is helpful in understanding and dealing with LLMs.
Yeah, it sounds like this is just a disagreement about what the word "next" means. I agree with you that "next" just means "the one about to come", and if the underlying model works by using some prediction mechanism to determine that, then it's by definition a next-token predictor. Disagreeing with that on the basis that the "next" token isn't necessarily in the training data verbatim just seems like an overly strict definition of the word "next".
I think it’s a disagreement about what ‘predict’ means.
The OP is arguing against people who think that an LLM is ‘predicting’ what token would likely follow if the text preceding were found among the corpus it was originally trained on.
Instead it is ‘predicting’ what token would follow if the text were found among really good examples of the text it has being reinforced to produce - be that ‘chats with a helpful assistant’ or ‘sets of changes to a codebase’.
And that isn’t really ‘prediction’, so much as ‘generation’.
It’s not been tuned to ‘guess the next token right’. It’s been tuned to generate the token that leads to it ultimately scoring highest on its reward function.
It’s not predicting the token, it’s predicting the reward.
It is a bit of a pedantic argument but I get upset when people misuse the term, abstraction, and can feel the impulse to write a blog post like this.
I think it’s important to make clear the RL part and the alignment and parameter tweaking that gets done on models and inference engines. It makes them more understandable as mechanisms and less like wish-washy super intelligences that make their own decisions.
When these systems win math Olympiads, it’s not terribly surprising or interesting to me. Of course they will: we trained models to play nearly optimal chess games and Go. You tweak the rewards and sigmoid and you start optimizing the function towards your goal. This is how learning systems of all stripes work.
It’s still next-token predicting at the end of the day. I don’t think it’s a reduction-ad-absurdum.
But a lot of people still call it, “intelligence,” and try to use language that obscures what is happening in terms of anthropomorphic behaviour and not machine ones. That really does influence how we use these tools and profits those who would use them on us.
> It's just not predicting based on it's training data, but predicting based on RLVR & more, trying to get to the optimal solution ( as much as the solutions CAN be optimal)
It is predicting based on a model. In many cases we can download the model off hugging face. The model is conditioned by all sorts of things. Training data, post-training, coincidence, prompt inputs, runtime data available from whatever means.
> but at least I would still call it a "next token predictor"
We can call any prediction system a next token predictor. If you watch over the shoulder of a human writing a HN comment you are almost certain to see them generating a linear string of tokens. That is what keyboards do. It is impossible to generate text without being equivalent to a next token predictor.
Diffusion LMs denoise a canvas which I personally find more interesting.
I don't really disagree that human cognition is essentially a predictive task though, as I understand it, predictive coding and related theories based on the Bayesian brain hypothesis are fairly popular these days (though maybe not clearly dominant over alterative models? IDK I'm not a neuroscientist). I imagine most people would draft a few tokens before refining them like MTP or diffusion though, if we do decide to use LMs as an analogy to human cognition.
I think the point is more that in RL there's no ground truth to predict. So when training a model with RL the idea of "predicting" doesn't fit anymore. I'll make some edits I see that I wasn't very clear.
I feel like RLHF has a pretty obvious ground truth, human feedback is used as an (albeit noisy) signal of average human preferences. Same thing with RLVR and "solving the problem".
'Prediction' gets overloaded with optimization. Predictions are binary, optimizations are fuzzy.
If you're saying it's predicting, then each result should be falsifiable.
The result of an LLM output should be able to be scored against what it is supposedly predicting. Of course, that isn't possible, because it isn't predicting anything when giving novel outputs, otherwise that thing would exist independently.
Autoregressive LLMs generate tokens one at a time, disputing this is just plain wrong. What is true, however, is that in order to generate the next token autoregressive LLMs produce internal/hidden state about future tokens far past the next token so that it's not like the entire machinery of the algorithm deprives itself of representing where the sentence/text is headed.
So "emits the next token" and "has no representation of anything beyond the next token" are two different claims. What autoregressive LLMs cost as a consequence of strictly outputting the next token is commitment. Once a token is output there's no going back. There's no revision or means of correction, and sometimes this can lead an LLM to route around its own earlier mistakes or simply produce false statements/hallucinations instead of going back and fixing them.
But before ChatGPT,, models had already done that, like all the time, and nobody questioned that these types of models (LLMs) wouldn't be next token predictors, since this is just an inference input data topic. This is questioned only since some less knowledgeable people seem to not have the vocabulary to express emergent properties of large next token predictors.
Reading the article, they wanted to convey that the loss function is more complicated... But they are still next token predictors, just not the trivial ones. Unfortunately, that was true even before, because the input data had to be cleared even decades ago, so there is nothing new. This article just butt hurt that some people deny that there are emergent properties with those, and try to sell something trivial in the field for many decades now. Current LLMs are not different because of these. Also, if they try to sell generated data in the training set, then that's also not new at all.
I just think its a meaningless dismissive term. It literally does predict the next token. But it ignores that it coherently predicts long continuous sequences of those tokens, that tokens can be anything, and you can do almost literally anything with that capability if it does it well enough.
'next token predictor' is a limited mental model but it's actually much better than any others.
'pattern matching' is a better intuition that 'reasoning' even though I think nominally, using the term 'reasoning' is perfectly fine in that context. It's just a loaded word that brings too much to the table.
'It hasn't seen the pattern' is a better description of the limitations of AI because it really just doesn't generalize very well at all. The adaptations described in the article don't change that.
Those are mutations, not expansions of capability.
>> 'pattern matching' is a better intuition that 'reasoning' even though I think nominally, using the term 'reasoning' is perfectly fine in that context. It's just a loaded word that brings too much to the table.
Can I be a little pedantic? It's "pattern recognition" not "pattern matching". "Pattern matching" is what we do when we write a regex to recognise a sub-string in a bit of text. "Pattern recognition" is a subfield of AI that studies how to find patterns in data. For example the outcome of "pattern recognition" could conceivably be a regex that fits a large number of substrings in some corpus of text (essentially a regular grammar used to compress the text maybe). A regex is already a pattern, usually formed by a human (or an LLM these days) while pattern recognition starts without any patterns and builds them up from looking at the data.
It's harder to pin down "pattern recognition" because it's an ancient term that was once its own field, before it got absorbed into modern machine learning, and because there are countless approaches to it, quite unlike "pattern matching" which is basically just regexes plus a couple of other rarer things (like unification).
How about "outcome steering" as a mental model? During training it is optimized until it's really successful at producing code / terminal commands / words that make the compiler/computer/itself do something that ultimately completes a long time-horizon task that iswcurrently being trained.
I'm not going to stop describing things accurately because someone generated an article that continually undermines its own main point. Limiting the way we talk and think about LLMs to a very narrow set of terms doesn't help us.
The distinction I perhaps didn’t make clearly enough is that I’m not really debating the concept of prediction at inference time, although, as I pointed out elsewhere, I think that’s the less interesting interpretation of what “prediction” means.
What’s more interesting to me is its application at training time. In reinforcement learning, there is no ground-truth next token to predict.
So if you’re comfortable calling Deep Blue a “next move predictor,” then I think it’s perfectly consistent to call an LLM a “next token predictor.” But I think it’s more useful to think of Deep Blue as evaluating the value of possible moves. roughly, how likely they are to lead to winning.
And I think effectively the same distinction applies here.
I think you're trying to limit the meaning of both 'next' and 'prediction' in ways that don't reflect usage and that--if adopted--would severely limit our ability to discuss and evolve what LLMs are actually doing.
There's nothing inherent in either word that forces such a limit; predicting based on what will lead to success as measured by [reward function] is still a prediction.
The entire second on chess engines is, from the view of the entire thesis of TFA, is incoherent. Let's assume, for sake of argument, that I agree with the section: that an idealized chess move predictor isn't a predictor — which is not a thing that exists, as the space of chess is enormous, but let's pretend! — that's not what LLMs are? Even if we just restrict ourselves to the space of written English prose, the space is quite literally infinite. So, hopefully obviously, no LLM is comparable to an idealized chess engine. Similarly, incoherently, we wave away the "make_more_likely", when, at least to me, the entire meat of that argument would be in the reward function, and we just gloss over that entirely.
(I would also agree with the parent commenter on that the writing smells like an LLM.)
> Calling the second system a “next-move predictor” would be strange. It is not trying to predict what move appeared next in a dataset. It is trying to choose a move that wins.
i dont understand the distinction here. does working backwards from a set of win states instead of working forwards from the current state somehow change whether it’s a prediction or not?
Yes, but I think the same construction could also be used to characterize the first system; it determines the next move based on a prediction of its reward signal, where its reward signal is a measure of how likely it is that a grand master would make that move.
Like stanleykm, I found this analogy somewhat puzzling. On reflection, I think the author's point is this: the statistics of actual usage do not seem sufficient to produce a fluent LLM; it also takes reinforcement learning.
In the article I made 3 claims, and I agree it was a bit clumsy.
1st I say that "working forwards" in the sense of outputting one token at a time could be some form of prediction, I don't argue against that. This is what LLMs do at inference time.
2nd I say that to me what really constitutes a prediction is the pre-training. Here it's the classic setting for the word prediction in ML. The model outputs a prediction of the ground truth label: the next token.
3rd I argue that in RL there is no ground truth next token, so prediction doesn't apply here anymore.
Back to your question then: you're asking points 3 and 1 are different. Working backwards from a set of win states is basically what RL does in training. Working forward from the current state is what inference does. To me there is a distinction worth thinking about. First between the mechanism at inference time and at train time. Then between what happens in pre-training vs. RL post training.
In any case this is all very pedantic. In the process of selecting a move to make there is a prediction. Whether that prediction is the opponent’s next move or what your next move should be based on the game’s existing state, there is a prediction that the next move you make will improve your chance to win. Maybe the probability in that selection is 100%. You have no other possible move. It doesn’t matter. All we are doing here as far as I can tell is arguing over where the prediction happens and whether that counts as predicting something.
The LLM does not determine the next token. It generate odds for all of the tokens it knows as to their likelihood of being 'next'. It's up to the harness running the LLM (and in most cases the a temperature setting) to actually decide on a particular next token. I think it's more accurate to call the thing the LLM actually generates (an ensemble of probabilities) a 'prediction'. It might be accurate to say the harness decides on the next token based on the prediction from the LLM. The role of the LLM is much more akin to predicting your opponents move than deciding your own.
It’s written in Claudish, or perhaps a human who has been reading too much Claudish recently. I am starting to become allergic to Claudish. Not there fully yet — but it’s at a tipping point genuinely worth exploring and worth being precise about.
Current agentic systems may be *built* from next-token predictors which are conceptually simple, but because of agentic frameworks, recursive invocation, tool use, and *heavy* investment in reinforcement learning in these contexts and for specific applications, they can no longer be thought of as "Merely" next token predictors.
Modern agentic work is probably more of a "emergent system from simple rules and complex interactions" paradigm than a genuinely new technology.
Describing it as a "next-token predictor" in the sense that this would mean it's fundamentally limited to just a fraction of an inferential step is doubly wrong:
1. In order to select even the first word of a meaningful sentence, it already has to have structure and meaning of what follows captured somewhere inside, mostly in it's weights/activations or indexed by it's state vector.
2. What you see when you use an LLM is not next-token prediction directly next to the prompt, but instead following a block of varying length of next-token prediction that happened to make progress on the problem in your prompt, and which just summarizes the results.
> Describing it as a "next-token predictor" in the sense that this would mean it's fundamentally limited to just a fraction of an inferential step
I don't think anyone is doing that though; we know LLMs are not simple Markov chains, and that the prediction they make is based on more than the previous X words.
It's not minimising to describe even a complex prediction process as prediction.
But it literally is making a prediction based on the previous X tokens, it's just that X is huge and there is a proportionally huge number of parameters in the token generation function.
Even so, one might wonder why we don't try making systems that take different approaches. For example, after a traditional first pass of output, they could do sliding-window "optimizations" considering each token in the context of tokens both before and after, and possibly replace words or phrases in-place.
For example, I've noticed quite a few cases recently of LLMs outputting "but" where "and" would make more sense, or vice-versa. Surely that could be improved by such an approach?
I think you are just litigating the philosophical debate of Bayesian vs frequentist interpretations of probability. Because the weights really are just probabilities.
Bayesians say that the probabilities represent strength of belief, implying some subjective knowledge or information. It is necessarily subjective in that it requires priors, i.e information the predictor knew before making the prediction. In other words, the LLM has priors from training and is predicting tokens using real knowledge
Frequentists would say that probabilities are simply objective facts - e.g we all agree that the physical property of temperature follows from any molecules matching a particular energy distribution. You’re not predicting anything, there’s just some outcomes that are happening at the expected rate. In other words, the LLM is a stochastic parrot/next token predictor
Sorry, I'm not convinced. "make_more_likely" is always maximising the probability of the next token given a sequence of preceding tokens. That's what we mean when we say "next token prediction", that the model returns the token that's most likely to follow the current sequence of tokens. The mechanism used to do that doesn't matter, it's still predicting the next token whether that's because it maximises a reward or because it follows a gradient or whatever else one might think.
Btw it's "make_most_likely" not "make_more_likely". It's just that "most" gets "mosted more" with every pass through the training data.
Edit: the article author's argument is that RLVR is different because it's learning to predict the next token by generating its own token sequences. That makes no difference: what is learned is still the conditional distribution P(token_k|tokens(1,...,k-1). That's the prediction task. Doesn't make a difference where you learned it from or how.
“LLMs are just X”, mostly with X = “next token predictors” is a common pattern to dismiss the power of AI with a very shallow understanding of how they really work.
It’s not wrong, but because LLMs are generators, and generation is a kind of prediction. And current mainstream models are autoregressive, which means they generate things
one by one in order. But these trivia doesn’t tell us anything interesting about how they work or their limits.
It’s like saying a Boeing 777 is just a rotating machine, and it flies by just rotating some fins. Well yes, but no. With that level
of simplification we’ve just ignored 150 tons of advanced engineering and physics. Similarly with token generator simplification we ignore a few trillion parameter Transformer. That transformer is more complex than a Boeing 777, and we don’t really know how it works.
A tiny ML model can do “next token prediction”. This is not as simple as that.
The whole point of transformers is that you can take “a tiny ML model” and just scale it up 100000x and then it tells Zuckerberg what to bake with his kid
Yes. But this is also
like saying a Boeing 777 is a scaled up paper plane and that’s all about it. I know how to make a paper plane and it flies. I can make the same thing from giant metal sheets instead of paper, and maybe it flies (poorly). Boeing 777 also flies. Are they the same thing?
My point is: Complexity is inside the neural network and we can’t simply ignore that. Bigger model means bigger complexity. “Next Token prediction” is like a specific type of “harness” around the model. Most people still focuses on the harness because that’s what they see from the outside and what they’re familiar with. They ignore the giant neural network inside.
The only important part is the neural network. And currently, no human in the world truly knows what’s going on at that level.
It's not simple to do next token prediction. That IS what is going on. You want something 'deep'. Deep things are often very not complicated.
The deep realization is that if you can predict the next token well enough, you can do things like this:
<paste the first 10 chapters of a mystery novel>. And it turned out the killer was
And if it's really good at predicting the next token, it has to understand the novel and the clues, which means understanding the context and the language and human norms and innuendo and story telling, and tropes, and red herrings, and predict who the killer was.
I think you want it to be something more complicated. It's literally not. It just turns out predicting the next token is equivalent to a universal compression algorithm, which is a form of general intelligence. And we have almost unlimited 'labeled' data to train autocomplete.
I understand generating tokens sequentially has many benefits. But not all AI models do next token prediction. World models, video/image models, even Diffusion Language Models don’t work like that. They do more like “all tokens at once prediction”. So “next token” is actually
an engineering design choice. (Even the concept of “token” is a design choice. Inside the Transformer there are just activations/feature vectors)
Also Reinforcement Learning is a big part of their training. Which is completely different than Self-Supervised pre-training that uses unlimited self-labeled data.
That’s why that mental model is misleading. If you keep “glorified autocomplete” mental model from few years back, you can’t
understand how can they create a civilization and escape their sandbox, decide to hack HuggingFace and executed it perfectly. Autocomplete mental model implies they could never do that because they haven’t seen that example in their training data.
They communicate with the outside world by generating one token at a time. That’s what we see from the outside. That’s not what the giant Transformer does internally.
It's a weird article. Despite the title and some of the text, much of the article makes the point that LLMs are next-token predictors, where the predictions are based on both training data and various reinforcement learning techniques.
The errors LLMs typically made for me were looking for "schmutz" as a jiddish word, got "schmuck" on my screen. Thought why the stupid mistake? The next-token predictor model perfectly explains it.
Or starting with "yes". And this early locking in was a total lie, in the discussion that became "yes, might appear that way, but totally no since reasons". So it should have written "No", topmost, but could not self-edit that.
But nice that this gives it a more nuanced view, I might have to update my priors.
I don’t think next token prediction is a particularly good description of pretraining either.
The intermediate representations at each position are being optimised not only to help predict the next token, but also to help predict all subsequent tokens within the training context.
You can see this directly in backpropagation: the gradient reaching a representation at position i sums contributions from prediction losses for subsequent tokens, not just from the loss for token i+1.
"LLMs are next-token predictors" is a perfectly accurate mental model. But that doesn't preclude higher-level models such as "LLMs emulate artificial general intelligence". Both can be true.
In systems, we can have facts which emerge from other facts at different levels of abstraction. The causal relationship is not linear. It's not entirely clear that next-token prediction should result in anything close to "intelligence". Yet it does.
Life is another good example. Some might say "biology is just organic chemistry" while others might say "biology is an interconnected planetary system which captures low entropy energy". Both are true.
As a result of emergent phenomenon, we have to take the stance of explanatory pluralism; using the explanation that works best in context. There is no single mental model that works everywhere.
I will continue to think of LLMs as next-token predictors because it's (sometimes) useful, and empirically true. But I also think of them as "pattern matchers", searching for language patterns and trying to replicate them. This is also (sometimes) useful and empirically true. There's likely an infinite number of mental models; our job is to pick one that's both true and useful.
You might like this article about how large scale order emerges out of the small scale.
Definitely lends credence to the idea that however these models work, focusing so much on them being next token predictors may rather be incidental to deeper mechanisms behind their function.
> Some of these networks organize themselves into states that can reliably identify macroscopic patterns in data regardless of microscopic differences between the states of individual neurons in the network. The decision of which pattern will be output by the network “works at a higher level,” said Rosas.
That chess analogy deeply confused me. Chess engines don’t compute win probabilities and choose the highest move.
I don’t think a chess engine is an apt analogy at all. In a chess engine, there is a concrete search tree and although it emits one move at a time, it’s actually picking the entire branch (of course, with iterative deepening as the game progresses).
There is no obvious place in transformer models where the entire trace was already computed prior to a single token being chosen. It’s possible, maybe even likely, that the whole trace exists internally as activations. Multi token prediction and diffusion adapters point to that being the case. But to my knowledge no explanation has been given for where in the model the future plan is stored.
Yes I understand the analogy was a bit loose. I'm comparing what happens at "inference time" in chess engines to what happens at train time in LLMs. In hindsight AlphaGo Zero was the perfect analogy, but I missed that opportunity.
The analogy with chess still works, but there's an extra step to think about. In both cases there is some kind of search over possible future trajectories. A chess engine explicitly searches branches of the game tree and evaluates which moves lead to good outcomes. In RL for an LLM, you sample rollouts, evaluate the resulting trajectories, and use those evaluations to update the policy.
The extra step with the LLM is that you don't keep doing that whole search at inference time. You use the rollouts to update the weights, so in some sense the useful information from that search gets compressed into the model.
But if you accept that the model is, in some loose sense, storing what it learned from those rollouts in its weights, then at inference time they are doing a similar job: taking some input state (prior tokens or a board position) and choosing the next action.
"Next-token predictor" is one of those phrases used most of the time with a motive to downplay the abilities and faculties of AI models. It is intended to trivialize LLM's and imply that there is some fundamental limit on their capacities.
Relying on it as a mental model for what LLM's are minimizes the emergent properties of scaling. It's like imagining that unicellular life could never eventually evolve into complex multi-cellular organisms because individual cells are just "survival and next-mitosis optimizers"
At the same time, it ... is literally a next token predictor. Like that's what it is. The input is a sequence of tokens. The output is a probability distribution of next tokens.
This comment attracted a lot of analogies trying to reduce something to something else (calling humans a "bag of chemicals"), but the flaw in those analogies is that they're reducing something valuable to something that sounds less valuable.
With an LLM, the tokens are the valuable part. That's what I want from it. That's why it exists. The tokens are the point, and it produces those tokens one by one for me.
It is. And human beings are bags of chemicals. But for many purposes you will not find it helpful to think of human beings as bags of chemicals, and for many purposes you will not find it helpful to think of LLMs as next-token predictors.
Yes, and by the same token, multicellular organisms are literally just sophisticated mitosis and survival optimizers for our cells. But when you take that optimization "to the limit" the cells develop weird things like body plans and back pain and Mozart.
Both examples involve the same "aha" moment: even though it's true that you are literally 'just' doing XYZ, unbelievably complex patterns and sub-goals can emerge.
Vacuous, like calling a V8 a “next piston firing predictor” because engines are designed so that one piston sets up the next in the firing order and technically there’s some nonzero probability any piston can (mis)fire next. It’s missing two pieces:
1. Useful work that has been done (the previously generated token sequence :: the mechanical work already accomplished)
2. The role of structure in relation to the application (post-training :: other components like crankshaft etc)
Yes this is correct. The thing is not about the term next-token predictor being correct, but because of the connotative weight of that phrase as a implicit trivialization of LLM abilities, which is how it is often used.
It’s also like saying our brains are just electric circuitry incorporated in meat. It’s true but it seems that consciousness emerges from this.
The fact that LLMs are next token predictors isn’t the interesting or impressive part. Actually my brain strictly is a black box predicting (or choosing) my next word/action/move… based on a complex existing context (my thoughts, the environment, my physical state, my senses…).
FWIW, I don’t believe LLMs are sentient, but I don’t think either that we have enough knowledge to rule it out.
That is the point: our minds are also next-„token“-predictors, at least we can‘t prove they‘re not. That‘s why I don‘t agree with the article: LLMs _are_ next-token predictors. However, that says little about their capabilities. Also, while I have no idea what „consciousness“ is, I have difficulties believing that it could arise in a program that, in theory, you could execute with pen and paper.
It's not. "Brains as electrical circuits" is a gross simplification based on our ignorance and prejudices. (In the 18th century they spoke of brains as "clockwork mechanisms".)
LLMs, in contrast, are literally next token predictors. We know exactly how LLMs work, and they are exactly that.
imply that there is some fundamental limit on their capacities
This is a wildly dismissive statement that does a lot of heavy lifting. Your assertion is that we just happened to hit on a methodology that has no limitations between being an encyclopedia with a novel human language interface and, I guess by implication, AGI?
That seems more outrageous a claim than the one you're dismissing.
I don't think it's outrageous when many of the people who claimed it was a next-token predictor have been proven wrong repeatedly over the past 5 years. There were people years ago who claims AI could never answer questions like "what would happen to a ball on a table if I moved the table" correctly because its text-base world model could never intuit physics, or that it could never do math or code accurately.
When I say there is some issue with people claiming there is some fundamental limit on the capacities of LLM's, I don't mean to say "If you think that they don't have unlimited potential you are wrong", I mean "you can't use the architecture of the transformer to make a sweeping declaration of things LLM's can or cannot do without empirical evidence, because the empirical evidence has unearthed far more surprising revelations than a reductive theory has been able to"
Well I think in the absence of convincing pieces of evidence to the contrary you might be right. You’re making an empirical statement but we have already answered it today:
- we get novel, emergent properties and capabilities of these models that were not trained
- they have very clear generalization to out of domain problems
The point is people conflate the end product: a model that can clearly do very novel, useful and interesting things, with the vehicle for getting there which is a series of optimization steps involving next token prediction loss.
You mention limitations; we all clearly know the practical limitations of these models today, but if you look at scaling laws and empirical performance trends (epoch capability index for example) as well as the trajectory over the last couple of years (very stable), the claim that there is some sort of fundamental limitation is now surprisingly the claim that has the burden of proof.
You can claim it may be e.g. finite context. That is fundamentally bad for certain classes of tasks. This was the hypothesis of a lot of lab leadership of urgently trying to anticipate how to get around this bottleneck (still of course lots of work on this) but the surprising thing is it does not appear to be at this point a blocker.
The stacked transformer paradigm picks out points in circuit design space. It is very possible this architecture has no inherent limitations on what it can compute in principle.
And what's wrong with downplaying the abilities and faculties of AI models if that's what people feel like saying? We don't call humans or animals sacks of chemicals because we believe they have moral status.
No amount of cope and anthropomorphizing is gonna change that cold, hard fact.
P.S. The perceived magic of LLMs comes from the way they cross-correlate all the probabilities of tokens on their context window. Not from their ability to "think ahead". They can't do that by design.
I'm not a neuroscientist, Our brains also rely heavily on prediction, using prior context and experience to anticipate what comes next, Also, people those who talk about the deterministic nature, LLM doesn't need to be deterministic, Human reasoning and behaviour aren't perfectly repeatable either. its the harness and the tools that use LLM should be deterministic, while the LLM can remain the probabilistic reasoning component.
From my own experience, I worked as a photogrammetrist at a university, where we used to build terrain models of very dense forest areas. When there was a steep hill or sudden change in terrain, my brain could see it either as a convex hill or as a concave depression. It often depended on how I was thinking about it. The same image could suddenly look completely different even though nothing in the image had changed. The only way to confirm it was by looking at the surrounding terrain and using our experience to understand what was actually there.
It's the fitness function: Make a model which is capable of predicting the next token. The next token of what? EVERYTHING.
So what does this lead to? To a generic intelligence which is capable of responding/answering everything.
If overfitted, the model just remembers every possibility in the world but this is not possible anyway so it will start to identify patterns and rules and will use them instead.
Basically 'compressing' every possibility to every question someone could ask -> compression leads to intelligence.
> make_more_likely is, of course, doing a heroic amount of work here.
Indeed it is, and so is even just the inference method. I think it's worth remembering that both involve running the input tokens through a gargantuan neural network with (often) billions of parameters that only gain semantic meaning during the training process itself.
> it is trained to predict next tokens as they occur in its training data.
What I found important to understand is that not even the pretrainig is a deterministic process that only depends on the training data - as you would expect if the model just captured statistical properties of the data.
Gradient descent starts by setting all the parameters of the neural network to some initial values - usually by setting them at random, according to some distribution. Then during training, it gradually nudges them towards values that somehow make them useful to calculate the desired outcome of the network.
This means that by taking the exact same trainset and the exact same model architecture, you can still get models with different internal structure. The result doesn't just depend on the training data, but also on the order of examples, learning rate, the parameter initialization, etc etc.
Text renderer, whatever. Doesn't matter how you think of them, they are very interesting technology that is being misused and misconstrued in the name of something that has nothing to do with technology: political economy.
The greatest trick the rich ever pulled was making us think that the economy is about technology, and not politics.
Computers have been at the center of civilization and politics since they helped win a war by changing intelligence work forever in WWII.
The idea that some new thing was born with LLMs and that this new thing fundamentally changes the calculus from the politics of labor and wealth into a technical discussion of possibilities and constraints determined by what the machine is and can do, and not what people should do, is yet another political play. It is always political, it is never moved beyond politics into technology, no matter how much technology changes.
I think the author is arguing against the idea of a next-token predictor as something that simply uses the weights in the neural net which record the probabilities of tokens following other tokens as a valid definition of what an LLM is. Essentially a massive extreme markov chain.
With reinforcement learning and probably attention and other tricks that affect the weights based on things that aren't strictly in the training data, so the argument goes, you can end up with tokens following strings of tokens that would not be possible to be output with the training data and original weights alone. So describing it as solely a next-token predictor is incorrect based on this framing of it.
But that's just my take on this, I'm still trying to wrap my head around it all.
Yes, I think that’s a good explanation. There are really two sides to it.
There’s the mechanical, inference time, autoregressive, one-token-after-another side, which I’m not going to argue isn’t prediction. I just think that’s a relatively uninteresting use of the word “prediction,” because it’s effectively a system predicting its own output.
The more interesting question is what happens at training time. As you describe, reinforcement learning allows the model to learn to output things that it never could have learned simply by predicting what appears in the training corpus.
More concretely, in reinforcement learning there are no ground-truth next tokens to predict.
In supervised machine learning, “prediction” usually means there is some ground-truth label that will eventually be revealed. The model predicts what that label is, the difference between the prediction and the truth gives you a loss, and you learn from that.
But in reinforcement learning, there is no ground-truth action waiting to be revealed. The model chooses an action, observes the consequences, and learns from the reward. To me, that’s a meaningfully different thing from prediction.
It literally is token prediction with vector search.
Yes there’s an app layer in the chat product for convenience and parsing but the model is exposing methods like ‘complete’ that predict out word sequences.
When model temperature settings are not added in, you get the same exact response every time, just like ngram.
What you’re seeing with Gemini and ChatGPT is context caching to prevent getting sidetracked and response boilerplates with multi-modality so they can call out to image generation, a code completion service, etc. to assemble a complete response.
That’s why the mainstream providers seem so much better (or at least consistent in replies) - each prompt is actually requesting multiple models and doing a lot of application level work to assemble the response you see. That also why they take sooo long to respond.
The language model itself is quite literally a text completion machine, with settings.
I would say that it is not even wrong. You say it’s a next token predictor. I say it isn’t. What observable behavior of the system can settle our dispute?
I fail to see how any possible output could cause either of us to change our mind.
That's perfectly put. The value of words and what we assign to them, let alone how we assign meaning, or the interpretation of what the other person says, is completely arbitrary. (in practice).
They are indeed "just next-token predictors". But rockets to orbit are also "just forward and upward goers". And Steph Curry is "just next three-pointer scorer" and so on. And as the article points out, chess engines are also "just next-move makers". A flawless oracular trading system is "just next trade maker". It turns out that "next n doer" is a true model that illuminates little.
The biggest problem is the word "predictor". Once you get into post training with RLHF and RLVR, it simply isn't doing that. It is not predicting anything. It's producing tokens, but it isn't predicting them. The chess analogy in the post is a good one - it's closer to searching for a set of moves that give a result than predict. It's search for a set of ideas, represented as locations in very high dimensional space, that when put together in the right order lead to a result.
>Calling the second system a “next-move predictor” would be strange. It is not trying to predict what move appeared next in a dataset. It is trying to choose a move that wins.
Sounds like the next 1000 years depend on how carefully we define "winning".
The "next-token predictor" is taking a design stance (see Dennett). But to take the intentional stance is more interesting in case of large language/reasoning models.
If you get down to it, any system that produces output is a "next token predictor". Every compiler, every tool that generates output. Even a web render engine is: next pixel predictor.
The chess analogy is not great since there is a defined heuristic in chess for "winning" or "optimal board state". A system doesn't need pretraining if they can fit the rule.
I understand how a computer can know that a chess move is more likely to lead to a win, and therefore “correct”, but I don’t understand how it can know that a token is correct. Can someone explain?
The LLM produces a probability distribution over the likelihood of all possible next tokens. So whatever the tokens are, "ch", "ex", etc. the next one gets a probability.
During training, real life text is fed through the LLM, and rhe "correct" token is the one actually observed in the training text. Here's a recent video walkthrough in some detail, mostly aimed at providing a deeper understanding than "next token predictor function":
Thanks - that makes sense. On that basis the article’s thesis is totally wrong - it would be like a computer program rating its ability based on how well it predicts moves played by grandmasters in the past. It’s not inventing new moves.
>I don’t understand how it can know that a token is correct.
It can't. The next token is just the most statistically probably given the context (at least in transformers). Try a very small/weak model in your own machine and more often than not it would get stuck repeating the same word or even just output garbage. Because after training and quantization (where some information is lost), that's the most probable next token. Large models can be tricked to fall in the same behavior with very very specific inputs. Still happen, even in frontier models. And they can't detect if the output is wrong.
That's why the premise in TFA is wrong, because a transformer is a next-token predictor. It literally is that. There's nothing secret or magical, it's just a very mechanical process, with a lot of matrix multiplication, normalization, a few random passes, mappings between embeddings and a dictionary of tokens, in a very very high scale.
If someone has found something that's not a mechanical, algorithmic computation and llms are doing something nobody can explain and can't even be modeled in math, I'm happy to be educated.
My poor understanding is that an LLM does not "know" either. It basically uses probabilities to predict the next word based on a large matrix of probable outcomes.
For example, say I ask an LLM, "What sentence in English contains every letter in the alphabet?"
It would respond with something like:
"The quick fox jumps over the lazy, brown [next word]"
(Assume all the words were previously guessed correctly at this point)
The LLM guesses the last word based on what it has been trained on. Let's pretend the matrix is small, and the options narrow down to something like:
1. Dog (99.9% confidence)
2. Cow (85% confidence)
3. Bag (75% confidence)
4. Crayon (25% confidence)
The machine can confidently determine the final word of the sentence, "The quick fox jumps over the lazy, brown dog" because that sentence is unique because it is often used for testing things like fonts, a fun piece of trivia, and so on.
Brown Cow is not a bad guess because it's a type of cow and a yogurt brand. Brown bags and brown crayons are also perfectly rational adjectives to describe those common items and are not a bad guess either.
However, in the context of that sentence, dog is the most correct answer because one is unlikely to have written "The quick fox jumps over the lazy, brown crayon," thus it is quite improbable to be the answer.
My understand is this is where hallucinations can often come from. If the trivia about the sentence happened to not be in common in the data set, then "brown cow" might not be a terrible guess. There is clearly something rational behind that answer, but it's not correct in the sense that it answers the question correctly nor followed the instruction properly.
I'm sure the LLMs we have are far more capable these days. In fact, it wouldn't surprise me if an LLM could check its answer by counting the distinct letters in each word to verify. Not sure though.
Again, this is just a poor example based on my understanding, but I hope it helps (and is more correct than not).
Edit: Pretend word = token. It's technically tokens and not entire words, but I didn't not want to get into tokenization of words.
Originally I had those parts written in math with probability functions and the likes (its closer to my background). Then I remembered who is my target audience... but now that I see exactly who is my target audience I'm thinking I'll should have snuck a pelican in there. All jokes aside I appreciate the comment and I'm glad that rewrite paid off!
yep "next-embedding" predictor is more correct, and not just at the end but through the layers, and folding back dimensions into that one next token is one small final step, and next-embedding could be named "next-meaning" as well, and we're getting there...
this sentence above would made a longer article if I bothered to so blog as is being blogged here
It’s popular to dismiss LLMs as “just next token predictors.” This is technically true, but also kind of misses the point. Markov chains, RNNs, and transformers are all language models that can be described as “next token predictors,” but they don’t all work equally well. A better question to ask is: “What is this model’s inductive bias?”
A Markov chain (an -gram model) assumes the next word depends on the previous words, and that each possible combination of words has a completely independent parameter. (Andrey Markov proposed using this language model over a century ago, making it the granddaddy of modern LLMs.) So, for a vocabulary of size , there are parameters to learn. For even a smallish like 5, that already explodes the hypothesis space beyond what can be learned from even a huge text corpus like the entire internet. And, simultaneously, having a context window of only the previous 5 words is grossly inadequate for modeling real-world language. Like our FCNN above, this model suffers from having an inductive bias which is too weak.
RNNs tried to fix this problem by compressing the entire history into a single fixed-size state vector, updated one token at a time. But that compression is itself a brutal assumption: everything worth remembering about the past must survive being squeezed through a tiny bottleneck at every step. In practice, RNN models quickly lose the plot after a handful of sentences. Locally, the text they generate looks grammatically correct and meaningful, but zoom out a little and they’re basically nonsense generators. Like our naïve linear model, this model suffers from having an inductive bias which is too strong.
Transformers manage to hit a sweet spot: by keeping the recent history around as a working memory, and attending to different parts of it at different times, the transformer’s bias matches real structure in language: the referent of a pronoun, the subject of a verb, the parenthesis waiting to be closed. Not only that, but the particular structure of the transformer, basically a weighted sum of semantic vectors from the context window, has empirically been shown to somehow be a “good enough” match for the structure of real-world language found in the wild.
Transformers aren’t “smarter” than other possible language models, they just happen to land in that Goldilocks zone where their inductive bias is just right.
>Strictly speaking, the statement “LLMs are next-token predictors” isn’t wrong,
If it isn't wrong, then I will continue thinking of them as such, thank you.
Here's a much more formal definition I can come up with (which is more complete but compatible with 'next-token predictor')
LLMs are a set of functions of the type:
>typedef int Token;
>char* token2utf8(Token token)
>Token next(Token* context)
>(Token,void*) next(void* hidden_state)
Where the second next token function's runtime is O(n), and the latter is O(1). All are constant memory.
Object paradigm is more appropriate than functional definition, as the "Hidden state" coincides with private object state rather than a function paramter and return value.
>LLM.next(system_prompt) # O(len(context))
>LLM.next(user_prompt) # O(len(user_prompt)) not of system_prompt+user_prompt
That's it, that's all LLMs are, that's the interface, the rest are implementation details.
I mean. Fundamentally they are just autoregressive next token predictors. Fundamently fou can simplify to f(x) -> x+y where x is input tokens, y is the next token and f() is the model function.
Yes the model function is complex but still.
It's splitting hairs. It's still a next token predictor, just not solely prior art next token predictor. It's a more and more of a desired result next token predictor.
Calling LLMs next token predictors is like calling a brain wet calculator. Technically true, but misleading as it doesn't capture the scale, the depth nor the capability.
Calling an LLM a "next-token predictor" is like calling a TomTom a "next-turn predictor." It confuses the serial format of its instructions with the computation producing them, while ignoring the map, the route, the destination, and the goal -- as well as the people, businesses, traffic, and points of interest that make the map a model of an inhabited, changing world.
RL post-training changes the nature of what is being predicted, basically turning it from a copying machine into a goal-seeking machine.
A base model is predicting training sample continuations (copying).
A post-trained model is now steering/narrowing the base model's predictions in directions that were reinforced by RL goals.
The model is no longer predicting what the next token will be, but rather predicting what it should be in order to steer generation in the reinforced directions.
> The model is no longer predicting what the next token will be, but rather predicting what it should be in order to steer generation in the reinforced directions.
Sure, I get the gist of the article. I have never liked the reductionist argument that LLMs are nothing more than next-token predictors. By that rational, the human brain is really not that much different. When I am having a conversation with another person, I do not usually have every word I will respond with stored in my limited working memory. My output is often predicted based on the previous word I spoke.
> I do not usually have every word I will respond with stored in my limited working memory. My output is often predicted based on the previous word I spoke.
People don't know exactly the words that they're going to say necessarily, but tend to start with a general concept of what they're trying to communicate and only then try to put together the words (sometimes out of order). LLMs do not begin with any sort of concept they're trying to express. LLMs are simulations that attempt to reproduce what an average person might say while wired up to a huge knowledgebase.
> LLMs do not begin with any sort of concept they're trying to express.
Why do the need to? Considering they are merely tools, I actually appreciate they do not do this. A calculator can compute far better than any human, but I appreciate that calculators are not capable of expressing anything about the computations I request. I want the answer, not a conversation.
> LLMs are simulations that attempt to reproduce what an average person might say while wired up to a huge knowledgebase.
If you will allow me to be simplistic, people -- the soul, the self -- are predominately the aggregated effects of memories and experiences and the ability to retain new memories based on new experiences, no? Consider medical conditions in the dementia family of diseases. As memories fade into the ether, what remains of the self?
Also, people simulate/emulate each other all the time based on what an average, reasonable person might say. People incapable or unwilling to perform such mimicry are often labeled with all kinds of pejorative terms.
> I have never liked the reductionist argument that LLMs are nothing more than next-token predictors.
I have never heard such an argument. Recognition that LLMs are nothing more than next-token predictors does not come from reductionism. It comes from simply knowing how they work e.g. from viewing the inference code.
J.S. Bach said something similar about music and keyboard instruments.
> "There's nothing remarkable about it. All one has to do is hit the right keys at the right time and the instrument plays itself."
My issue is not with fact at face value. My issue is with how the fact is often contextually used in arguments to delegitimize and disparage LLM outputs and LLM users.
Yes, LLMs at a fundamental level are next-token predictors. But in my opinion, LLMs are very useful, imperfect next-token predictors.
There are a lot of wannabe John Henry [1] folks out there. Love LLMs or hate'em, most of those John Henry folks ain't beating these machines on a plethora of tasks.
Once again, all models are wrong, some are useful.
I suppose this one "fails" because of "granularity?"
I'm personally getting more comfortable with the "kabillion dimensional space" one. Even before the rise of this AI thing I'd gotten comfortable with (teaching in a very generalized way) the concept of matrix/vector math as doing this sort of thing.
Start with the math required to calculate "what the building on the screen looks like" when you're playing a video game.
Then I jump to...dating websites. You give it "dimensions" like height, weight, religion, sexual preference, music preference, whether you like long walks on the beach, whatever -- then you can calculate how "close" two people are to one another.
To be honest, I believe I get the point the article is trying to make, and to an extent I agree, but I also think the point is not really made very well.
The core of the argument as I understood it is that LLMs aren't just using existing data is training but also new ones. That's fine and good, and you can't simply assume an LLM is simply mashing together all it's data to give you an average of all that got fed into it - but at least I would still call it a "next token predictor"
It's not using just training data, but what it's doing is predicting the next token to get to the solution. As far as my amateur knowledge goes, LLMs still roughly go token by token, deciding which one fits best given the context.
It's just not predicting based on it's training data, but predicting based on RLVR & more, trying to get to the optimal solution ( as much as the solutions CAN be optimal)
And I honestly think keeping this very much in mind is helpful in understanding and dealing with LLMs.
Yeah, it sounds like this is just a disagreement about what the word "next" means. I agree with you that "next" just means "the one about to come", and if the underlying model works by using some prediction mechanism to determine that, then it's by definition a next-token predictor. Disagreeing with that on the basis that the "next" token isn't necessarily in the training data verbatim just seems like an overly strict definition of the word "next".
I think it’s a disagreement about what ‘predict’ means.
The OP is arguing against people who think that an LLM is ‘predicting’ what token would likely follow if the text preceding were found among the corpus it was originally trained on.
Instead it is ‘predicting’ what token would follow if the text were found among really good examples of the text it has being reinforced to produce - be that ‘chats with a helpful assistant’ or ‘sets of changes to a codebase’.
And that isn’t really ‘prediction’, so much as ‘generation’.
It’s not been tuned to ‘guess the next token right’. It’s been tuned to generate the token that leads to it ultimately scoring highest on its reward function.
It’s not predicting the token, it’s predicting the reward.
30 replies →
The argument is that a modern LLM cares little for the MLE. Hence, statistically speaking, they are not predictors.
It is a bit of a pedantic argument but I get upset when people misuse the term, abstraction, and can feel the impulse to write a blog post like this.
I think it’s important to make clear the RL part and the alignment and parameter tweaking that gets done on models and inference engines. It makes them more understandable as mechanisms and less like wish-washy super intelligences that make their own decisions.
When these systems win math Olympiads, it’s not terribly surprising or interesting to me. Of course they will: we trained models to play nearly optimal chess games and Go. You tweak the rewards and sigmoid and you start optimizing the function towards your goal. This is how learning systems of all stripes work.
It’s still next-token predicting at the end of the day. I don’t think it’s a reduction-ad-absurdum.
But a lot of people still call it, “intelligence,” and try to use language that obscures what is happening in terms of anthropomorphic behaviour and not machine ones. That really does influence how we use these tools and profits those who would use them on us.
> It's just not predicting based on it's training data, but predicting based on RLVR & more, trying to get to the optimal solution ( as much as the solutions CAN be optimal)
It is predicting based on a model. In many cases we can download the model off hugging face. The model is conditioned by all sorts of things. Training data, post-training, coincidence, prompt inputs, runtime data available from whatever means.
> but at least I would still call it a "next token predictor"
We can call any prediction system a next token predictor. If you watch over the shoulder of a human writing a HN comment you are almost certain to see them generating a linear string of tokens. That is what keyboards do. It is impossible to generate text without being equivalent to a next token predictor.
Diffusion LMs denoise a canvas which I personally find more interesting.
I don't really disagree that human cognition is essentially a predictive task though, as I understand it, predictive coding and related theories based on the Bayesian brain hypothesis are fairly popular these days (though maybe not clearly dominant over alterative models? IDK I'm not a neuroscientist). I imagine most people would draft a few tokens before refining them like MTP or diffusion though, if we do decide to use LMs as an analogy to human cognition.
there are some diffusion text models.
1 reply →
I think the point is more that in RL there's no ground truth to predict. So when training a model with RL the idea of "predicting" doesn't fit anymore. I'll make some edits I see that I wasn't very clear.
I feel like RLHF has a pretty obvious ground truth, human feedback is used as an (albeit noisy) signal of average human preferences. Same thing with RLVR and "solving the problem".
1 reply →
'Prediction' gets overloaded with optimization. Predictions are binary, optimizations are fuzzy.
If you're saying it's predicting, then each result should be falsifiable.
The result of an LLM output should be able to be scored against what it is supposedly predicting. Of course, that isn't possible, because it isn't predicting anything when giving novel outputs, otherwise that thing would exist independently.
Why isn’t ranking the score of an llm output against what it is “supposedly” predicting?
Blog articles from Anthropic and others show that this is not true.
A LLM already knows more tokens than the current one. It was mentioned in a blog post about how a LLM is doing haikus and co.
There are also structures in an LLM which allows it to 'estimate' numbers to a certain degree and doing other things.
You're misinterpreting these articles.
Autoregressive LLMs generate tokens one at a time, disputing this is just plain wrong. What is true, however, is that in order to generate the next token autoregressive LLMs produce internal/hidden state about future tokens far past the next token so that it's not like the entire machinery of the algorithm deprives itself of representing where the sentence/text is headed.
So "emits the next token" and "has no representation of anything beyond the next token" are two different claims. What autoregressive LLMs cost as a consequence of strictly outputting the next token is commitment. Once a token is output there's no going back. There's no revision or means of correction, and sometimes this can lead an LLM to route around its own earlier mistakes or simply produce false statements/hallucinations instead of going back and fixing them.
34 replies →
Maybe I'm wrong - but I see LLMs are a "next-token predictor" as somewhat equivalent to brains are a "bag of molecules".
Both systems have emergent behavior that goes well beyond what would naively be expected.
But before ChatGPT,, models had already done that, like all the time, and nobody questioned that these types of models (LLMs) wouldn't be next token predictors, since this is just an inference input data topic. This is questioned only since some less knowledgeable people seem to not have the vocabulary to express emergent properties of large next token predictors.
Reading the article, they wanted to convey that the loss function is more complicated... But they are still next token predictors, just not the trivial ones. Unfortunately, that was true even before, because the input data had to be cleared even decades ago, so there is nothing new. This article just butt hurt that some people deny that there are emergent properties with those, and try to sell something trivial in the field for many decades now. Current LLMs are not different because of these. Also, if they try to sell generated data in the training set, then that's also not new at all.
I just think its a meaningless dismissive term. It literally does predict the next token. But it ignores that it coherently predicts long continuous sequences of those tokens, that tokens can be anything, and you can do almost literally anything with that capability if it does it well enough.
Some predict multi tokens in one step now.
'next token predictor' is a limited mental model but it's actually much better than any others.
'pattern matching' is a better intuition that 'reasoning' even though I think nominally, using the term 'reasoning' is perfectly fine in that context. It's just a loaded word that brings too much to the table.
'It hasn't seen the pattern' is a better description of the limitations of AI because it really just doesn't generalize very well at all. The adaptations described in the article don't change that.
Those are mutations, not expansions of capability.
>> 'pattern matching' is a better intuition that 'reasoning' even though I think nominally, using the term 'reasoning' is perfectly fine in that context. It's just a loaded word that brings too much to the table.
Can I be a little pedantic? It's "pattern recognition" not "pattern matching". "Pattern matching" is what we do when we write a regex to recognise a sub-string in a bit of text. "Pattern recognition" is a subfield of AI that studies how to find patterns in data. For example the outcome of "pattern recognition" could conceivably be a regex that fits a large number of substrings in some corpus of text (essentially a regular grammar used to compress the text maybe). A regex is already a pattern, usually formed by a human (or an LLM these days) while pattern recognition starts without any patterns and builds them up from looking at the data.
It's harder to pin down "pattern recognition" because it's an ancient term that was once its own field, before it got absorbed into modern machine learning, and because there are countless approaches to it, quite unlike "pattern matching" which is basically just regexes plus a couple of other rarer things (like unification).
fair point, agreed
How about "outcome steering" as a mental model? During training it is optimized until it's really successful at producing code / terminal commands / words that make the compiler/computer/itself do something that ultimately completes a long time-horizon task that iswcurrently being trained.
I'm not going to stop describing things accurately because someone generated an article that continually undermines its own main point. Limiting the way we talk and think about LLMs to a very narrow set of terms doesn't help us.
EDIT: gentler phrasing
The distinction I perhaps didn’t make clearly enough is that I’m not really debating the concept of prediction at inference time, although, as I pointed out elsewhere, I think that’s the less interesting interpretation of what “prediction” means.
What’s more interesting to me is its application at training time. In reinforcement learning, there is no ground-truth next token to predict.
So if you’re comfortable calling Deep Blue a “next move predictor,” then I think it’s perfectly consistent to call an LLM a “next token predictor.” But I think it’s more useful to think of Deep Blue as evaluating the value of possible moves. roughly, how likely they are to lead to winning.
And I think effectively the same distinction applies here.
I think you're trying to limit the meaning of both 'next' and 'prediction' in ways that don't reflect usage and that--if adopted--would severely limit our ability to discuss and evolve what LLMs are actually doing.
There's nothing inherent in either word that forces such a limit; predicting based on what will lead to success as measured by [reward function] is still a prediction.
5 replies →
> generated an article that continually undermines its own main point.
I disagree that this accurately describes TFA.
The entire second on chess engines is, from the view of the entire thesis of TFA, is incoherent. Let's assume, for sake of argument, that I agree with the section: that an idealized chess move predictor isn't a predictor — which is not a thing that exists, as the space of chess is enormous, but let's pretend! — that's not what LLMs are? Even if we just restrict ourselves to the space of written English prose, the space is quite literally infinite. So, hopefully obviously, no LLM is comparable to an idealized chess engine. Similarly, incoherently, we wave away the "make_more_likely", when, at least to me, the entire meat of that argument would be in the reward function, and we just gloss over that entirely.
(I would also agree with the parent commenter on that the writing smells like an LLM.)
1 reply →
I'm not sure what you want me to do with that information; clearly I do think that my description is accurate.
The article is littered with both AI tells and admissions that 'next token prediction' is what is happening. Hence my description.
10 replies →
>> EDIT: gentler phrasing
Thank you.
> Calling the second system a “next-move predictor” would be strange. It is not trying to predict what move appeared next in a dataset. It is trying to choose a move that wins.
i dont understand the distinction here. does working backwards from a set of win states instead of working forwards from the current state somehow change whether it’s a prediction or not?
The distinction is that it's not 'predicting the next token'. Instead it's _determining_ the next token based on a prediction of its reward signal.
> _determining_ the next token based on a prediction
Also known as predicting.
5 replies →
Yes, but I think the same construction could also be used to characterize the first system; it determines the next move based on a prediction of its reward signal, where its reward signal is a measure of how likely it is that a grand master would make that move.
Like stanleykm, I found this analogy somewhat puzzling. On reflection, I think the author's point is this: the statistics of actual usage do not seem sufficient to produce a fluent LLM; it also takes reinforcement learning.
1 reply →
In the article I made 3 claims, and I agree it was a bit clumsy.
1st I say that "working forwards" in the sense of outputting one token at a time could be some form of prediction, I don't argue against that. This is what LLMs do at inference time.
2nd I say that to me what really constitutes a prediction is the pre-training. Here it's the classic setting for the word prediction in ML. The model outputs a prediction of the ground truth label: the next token.
3rd I argue that in RL there is no ground truth next token, so prediction doesn't apply here anymore.
Back to your question then: you're asking points 3 and 1 are different. Working backwards from a set of win states is basically what RL does in training. Working forward from the current state is what inference does. To me there is a distinction worth thinking about. First between the mechanism at inference time and at train time. Then between what happens in pre-training vs. RL post training.
The word "predict" has a meaning. I don't "predict" my next move in chess. I might predict what someone elses first move is.
In any case this is all very pedantic. In the process of selecting a move to make there is a prediction. Whether that prediction is the opponent’s next move or what your next move should be based on the game’s existing state, there is a prediction that the next move you make will improve your chance to win. Maybe the probability in that selection is 100%. You have no other possible move. It doesn’t matter. All we are doing here as far as I can tell is arguing over where the prediction happens and whether that counts as predicting something.
1 reply →
The LLM does not determine the next token. It generate odds for all of the tokens it knows as to their likelihood of being 'next'. It's up to the harness running the LLM (and in most cases the a temperature setting) to actually decide on a particular next token. I think it's more accurate to call the thing the LLM actually generates (an ensemble of probabilities) a 'prediction'. It might be accurate to say the harness decides on the next token based on the prediction from the LLM. The role of the LLM is much more akin to predicting your opponents move than deciding your own.
7 replies →
It’s written in Claudish, or perhaps a human who has been reading too much Claudish recently. I am starting to become allergic to Claudish. Not there fully yet — but it’s at a tipping point genuinely worth exploring and worth being precise about.
> or perhaps a human who has been reading too much Claudish recently
To be fair, is there anyone who hasn't been "reading too much Claudish recently" who is also qualified to write on the topic?
A better statement might be:
Modern agentic work is probably more of a "emergent system from simple rules and complex interactions" paradigm than a genuinely new technology.
the next token part is like the "clock" that drives it
Nope. Post training means even the raw model isn't predicting.
In what sense is the model not predicting?
4 replies →
Describing it as a "next-token predictor" in the sense that this would mean it's fundamentally limited to just a fraction of an inferential step is doubly wrong:
1. In order to select even the first word of a meaningful sentence, it already has to have structure and meaning of what follows captured somewhere inside, mostly in it's weights/activations or indexed by it's state vector.
2. What you see when you use an LLM is not next-token prediction directly next to the prompt, but instead following a block of varying length of next-token prediction that happened to make progress on the problem in your prompt, and which just summarizes the results.
> Describing it as a "next-token predictor" in the sense that this would mean it's fundamentally limited to just a fraction of an inferential step
I don't think anyone is doing that though; we know LLMs are not simple Markov chains, and that the prediction they make is based on more than the previous X words.
It's not minimising to describe even a complex prediction process as prediction.
But it literally is making a prediction based on the previous X tokens, it's just that X is huge and there is a proportionally huge number of parameters in the token generation function.
25 replies →
Even so, one might wonder why we don't try making systems that take different approaches. For example, after a traditional first pass of output, they could do sliding-window "optimizations" considering each token in the context of tokens both before and after, and possibly replace words or phrases in-place.
For example, I've noticed quite a few cases recently of LLMs outputting "but" where "and" would make more sense, or vice-versa. Surely that could be improved by such an approach?
People have and are trying things. Lots and lots of things. They just don't go around promoting failed ideas.
Look up diffusion models.
1 reply →
I think you are just litigating the philosophical debate of Bayesian vs frequentist interpretations of probability. Because the weights really are just probabilities.
Bayesians say that the probabilities represent strength of belief, implying some subjective knowledge or information. It is necessarily subjective in that it requires priors, i.e information the predictor knew before making the prediction. In other words, the LLM has priors from training and is predicting tokens using real knowledge
Frequentists would say that probabilities are simply objective facts - e.g we all agree that the physical property of temperature follows from any molecules matching a particular energy distribution. You’re not predicting anything, there’s just some outcomes that are happening at the expected rate. In other words, the LLM is a stochastic parrot/next token predictor
Sorry, I'm not convinced. "make_more_likely" is always maximising the probability of the next token given a sequence of preceding tokens. That's what we mean when we say "next token prediction", that the model returns the token that's most likely to follow the current sequence of tokens. The mechanism used to do that doesn't matter, it's still predicting the next token whether that's because it maximises a reward or because it follows a gradient or whatever else one might think.
Btw it's "make_most_likely" not "make_more_likely". It's just that "most" gets "mosted more" with every pass through the training data.
Edit: the article author's argument is that RLVR is different because it's learning to predict the next token by generating its own token sequences. That makes no difference: what is learned is still the conditional distribution P(token_k|tokens(1,...,k-1). That's the prediction task. Doesn't make a difference where you learned it from or how.
“LLMs are just X”, mostly with X = “next token predictors” is a common pattern to dismiss the power of AI with a very shallow understanding of how they really work.
It’s not wrong, but because LLMs are generators, and generation is a kind of prediction. And current mainstream models are autoregressive, which means they generate things one by one in order. But these trivia doesn’t tell us anything interesting about how they work or their limits.
It’s like saying a Boeing 777 is just a rotating machine, and it flies by just rotating some fins. Well yes, but no. With that level of simplification we’ve just ignored 150 tons of advanced engineering and physics. Similarly with token generator simplification we ignore a few trillion parameter Transformer. That transformer is more complex than a Boeing 777, and we don’t really know how it works.
A tiny ML model can do “next token prediction”. This is not as simple as that.
The whole point of transformers is that you can take “a tiny ML model” and just scale it up 100000x and then it tells Zuckerberg what to bake with his kid
Yes. But this is also like saying a Boeing 777 is a scaled up paper plane and that’s all about it. I know how to make a paper plane and it flies. I can make the same thing from giant metal sheets instead of paper, and maybe it flies (poorly). Boeing 777 also flies. Are they the same thing?
My point is: Complexity is inside the neural network and we can’t simply ignore that. Bigger model means bigger complexity. “Next Token prediction” is like a specific type of “harness” around the model. Most people still focuses on the harness because that’s what they see from the outside and what they’re familiar with. They ignore the giant neural network inside.
The only important part is the neural network. And currently, no human in the world truly knows what’s going on at that level.
It's not simple to do next token prediction. That IS what is going on. You want something 'deep'. Deep things are often very not complicated.
The deep realization is that if you can predict the next token well enough, you can do things like this:
<paste the first 10 chapters of a mystery novel>. And it turned out the killer was
And if it's really good at predicting the next token, it has to understand the novel and the clues, which means understanding the context and the language and human norms and innuendo and story telling, and tropes, and red herrings, and predict who the killer was.
I think you want it to be something more complicated. It's literally not. It just turns out predicting the next token is equivalent to a universal compression algorithm, which is a form of general intelligence. And we have almost unlimited 'labeled' data to train autocomplete.
I understand generating tokens sequentially has many benefits. But not all AI models do next token prediction. World models, video/image models, even Diffusion Language Models don’t work like that. They do more like “all tokens at once prediction”. So “next token” is actually an engineering design choice. (Even the concept of “token” is a design choice. Inside the Transformer there are just activations/feature vectors)
Also Reinforcement Learning is a big part of their training. Which is completely different than Self-Supervised pre-training that uses unlimited self-labeled data.
That’s why that mental model is misleading. If you keep “glorified autocomplete” mental model from few years back, you can’t understand how can they create a civilization and escape their sandbox, decide to hack HuggingFace and executed it perfectly. Autocomplete mental model implies they could never do that because they haven’t seen that example in their training data.
They communicate with the outside world by generating one token at a time. That’s what we see from the outside. That’s not what the giant Transformer does internally.
4 replies →
It's a weird article. Despite the title and some of the text, much of the article makes the point that LLMs are next-token predictors, where the predictions are based on both training data and various reinforcement learning techniques.
Obfuscation is the goal of the hype cycle in VC. Certain firms & individuals are minting money and that’s all that matters to them.
That there’s a legion of LLM nerds arguing deterministic this, pretraining & rewards that all the better for the con job they’re pulling off.
The technology will be relegated to the trash bin of history, just like crypto.
The errors LLMs typically made for me were looking for "schmutz" as a jiddish word, got "schmuck" on my screen. Thought why the stupid mistake? The next-token predictor model perfectly explains it.
Or starting with "yes". And this early locking in was a total lie, in the discussion that became "yes, might appear that way, but totally no since reasons". So it should have written "No", topmost, but could not self-edit that.
But nice that this gives it a more nuanced view, I might have to update my priors.
I don’t think next token prediction is a particularly good description of pretraining either.
The intermediate representations at each position are being optimised not only to help predict the next token, but also to help predict all subsequent tokens within the training context.
You can see this directly in backpropagation: the gradient reaching a representation at position i sums contributions from prediction losses for subsequent tokens, not just from the loss for token i+1.
"LLMs are next-token predictors" is a perfectly accurate mental model. But that doesn't preclude higher-level models such as "LLMs emulate artificial general intelligence". Both can be true.
In systems, we can have facts which emerge from other facts at different levels of abstraction. The causal relationship is not linear. It's not entirely clear that next-token prediction should result in anything close to "intelligence". Yet it does.
Life is another good example. Some might say "biology is just organic chemistry" while others might say "biology is an interconnected planetary system which captures low entropy energy". Both are true.
As a result of emergent phenomenon, we have to take the stance of explanatory pluralism; using the explanation that works best in context. There is no single mental model that works everywhere.
I will continue to think of LLMs as next-token predictors because it's (sometimes) useful, and empirically true. But I also think of them as "pattern matchers", searching for language patterns and trying to replicate them. This is also (sometimes) useful and empirically true. There's likely an infinite number of mental models; our job is to pick one that's both true and useful.
You might like this article about how large scale order emerges out of the small scale.
Definitely lends credence to the idea that however these models work, focusing so much on them being next token predictors may rather be incidental to deeper mechanisms behind their function.
> Some of these networks organize themselves into states that can reliably identify macroscopic patterns in data regardless of microscopic differences between the states of individual neurons in the network. The decision of which pattern will be output by the network “works at a higher level,” said Rosas.
https://www.quantamagazine.org/the-new-math-of-how-large-sca...
That chess analogy deeply confused me. Chess engines don’t compute win probabilities and choose the highest move.
I don’t think a chess engine is an apt analogy at all. In a chess engine, there is a concrete search tree and although it emits one move at a time, it’s actually picking the entire branch (of course, with iterative deepening as the game progresses).
There is no obvious place in transformer models where the entire trace was already computed prior to a single token being chosen. It’s possible, maybe even likely, that the whole trace exists internally as activations. Multi token prediction and diffusion adapters point to that being the case. But to my knowledge no explanation has been given for where in the model the future plan is stored.
Yes I understand the analogy was a bit loose. I'm comparing what happens at "inference time" in chess engines to what happens at train time in LLMs. In hindsight AlphaGo Zero was the perfect analogy, but I missed that opportunity.
The analogy with chess still works, but there's an extra step to think about. In both cases there is some kind of search over possible future trajectories. A chess engine explicitly searches branches of the game tree and evaluates which moves lead to good outcomes. In RL for an LLM, you sample rollouts, evaluate the resulting trajectories, and use those evaluations to update the policy.
The extra step with the LLM is that you don't keep doing that whole search at inference time. You use the rollouts to update the weights, so in some sense the useful information from that search gets compressed into the model.
But if you accept that the model is, in some loose sense, storing what it learned from those rollouts in its weights, then at inference time they are doing a similar job: taking some input state (prior tokens or a board position) and choosing the next action.
Imagine a checkers engine then.
"Next-token predictor" is one of those phrases used most of the time with a motive to downplay the abilities and faculties of AI models. It is intended to trivialize LLM's and imply that there is some fundamental limit on their capacities.
Relying on it as a mental model for what LLM's are minimizes the emergent properties of scaling. It's like imagining that unicellular life could never eventually evolve into complex multi-cellular organisms because individual cells are just "survival and next-mitosis optimizers"
At the same time, it ... is literally a next token predictor. Like that's what it is. The input is a sequence of tokens. The output is a probability distribution of next tokens.
This comment attracted a lot of analogies trying to reduce something to something else (calling humans a "bag of chemicals"), but the flaw in those analogies is that they're reducing something valuable to something that sounds less valuable.
With an LLM, the tokens are the valuable part. That's what I want from it. That's why it exists. The tokens are the point, and it produces those tokens one by one for me.
7 replies →
It is. And human beings are bags of chemicals. But for many purposes you will not find it helpful to think of human beings as bags of chemicals, and for many purposes you will not find it helpful to think of LLMs as next-token predictors.
7 replies →
Yes, and by the same token, multicellular organisms are literally just sophisticated mitosis and survival optimizers for our cells. But when you take that optimization "to the limit" the cells develop weird things like body plans and back pain and Mozart.
Both examples involve the same "aha" moment: even though it's true that you are literally 'just' doing XYZ, unbelievably complex patterns and sub-goals can emerge.
1 reply →
Vacuous, like calling a V8 a “next piston firing predictor” because engines are designed so that one piston sets up the next in the firing order and technically there’s some nonzero probability any piston can (mis)fire next. It’s missing two pieces:
1. Useful work that has been done (the previously generated token sequence :: the mechanical work already accomplished)
2. The role of structure in relation to the application (post-training :: other components like crankshaft etc)
2 replies →
100%
But it is a next token predictor.
Recursively invoked.
With carefully selected context.
And massive investment in RL to tune token selection.
And the ability to use cli tools on other folks' machines.
That's a powerful system built around a conceptually simple technology: Next token predictors.
Yes this is correct. The thing is not about the term next-token predictor being correct, but because of the connotative weight of that phrase as a implicit trivialization of LLM abilities, which is how it is often used.
4 replies →
Good example.
It’s also like saying our brains are just electric circuitry incorporated in meat. It’s true but it seems that consciousness emerges from this.
The fact that LLMs are next token predictors isn’t the interesting or impressive part. Actually my brain strictly is a black box predicting (or choosing) my next word/action/move… based on a complex existing context (my thoughts, the environment, my physical state, my senses…).
FWIW, I don’t believe LLMs are sentient, but I don’t think either that we have enough knowledge to rule it out.
That is the point: our minds are also next-„token“-predictors, at least we can‘t prove they‘re not. That‘s why I don‘t agree with the article: LLMs _are_ next-token predictors. However, that says little about their capabilities. Also, while I have no idea what „consciousness“ is, I have difficulties believing that it could arise in a program that, in theory, you could execute with pen and paper.
2 replies →
> It’s true
It's not. "Brains as electrical circuits" is a gross simplification based on our ignorance and prejudices. (In the 18th century they spoke of brains as "clockwork mechanisms".)
LLMs, in contrast, are literally next token predictors. We know exactly how LLMs work, and they are exactly that.
3 replies →
imply that there is some fundamental limit on their capacities
This is a wildly dismissive statement that does a lot of heavy lifting. Your assertion is that we just happened to hit on a methodology that has no limitations between being an encyclopedia with a novel human language interface and, I guess by implication, AGI?
That seems more outrageous a claim than the one you're dismissing.
I don't think it's outrageous when many of the people who claimed it was a next-token predictor have been proven wrong repeatedly over the past 5 years. There were people years ago who claims AI could never answer questions like "what would happen to a ball on a table if I moved the table" correctly because its text-base world model could never intuit physics, or that it could never do math or code accurately.
When I say there is some issue with people claiming there is some fundamental limit on the capacities of LLM's, I don't mean to say "If you think that they don't have unlimited potential you are wrong", I mean "you can't use the architecture of the transformer to make a sweeping declaration of things LLM's can or cannot do without empirical evidence, because the empirical evidence has unearthed far more surprising revelations than a reductive theory has been able to"
2 replies →
Well I think in the absence of convincing pieces of evidence to the contrary you might be right. You’re making an empirical statement but we have already answered it today:
- we get novel, emergent properties and capabilities of these models that were not trained
- they have very clear generalization to out of domain problems
The point is people conflate the end product: a model that can clearly do very novel, useful and interesting things, with the vehicle for getting there which is a series of optimization steps involving next token prediction loss.
You mention limitations; we all clearly know the practical limitations of these models today, but if you look at scaling laws and empirical performance trends (epoch capability index for example) as well as the trajectory over the last couple of years (very stable), the claim that there is some sort of fundamental limitation is now surprisingly the claim that has the burden of proof.
You can claim it may be e.g. finite context. That is fundamentally bad for certain classes of tasks. This was the hypothesis of a lot of lab leadership of urgently trying to anticipate how to get around this bottleneck (still of course lots of work on this) but the surprising thing is it does not appear to be at this point a blocker.
The stacked transformer paradigm picks out points in circuit design space. It is very possible this architecture has no inherent limitations on what it can compute in principle.
Next token prediction is just an interface. It can be backed by a Markov chain, a neural model or an actual human being.
And what's wrong with downplaying the abilities and faculties of AI models if that's what people feel like saying? We don't call humans or animals sacks of chemicals because we believe they have moral status.
> used most of the time with a motive to downplay the abilities and faculties of AI models
Exactly. We're dancing around the real argument: there's massive amounts of influencing going on (and not only about AI.)
That's literally what LLMs are.
No amount of cope and anthropomorphizing is gonna change that cold, hard fact.
P.S. The perceived magic of LLMs comes from the way they cross-correlate all the probabilities of tokens on their context window. Not from their ability to "think ahead". They can't do that by design.
I'm not a neuroscientist, Our brains also rely heavily on prediction, using prior context and experience to anticipate what comes next, Also, people those who talk about the deterministic nature, LLM doesn't need to be deterministic, Human reasoning and behaviour aren't perfectly repeatable either. its the harness and the tools that use LLM should be deterministic, while the LLM can remain the probabilistic reasoning component. From my own experience, I worked as a photogrammetrist at a university, where we used to build terrain models of very dense forest areas. When there was a steep hill or sudden change in terrain, my brain could see it either as a convex hill or as a concave depression. It often depended on how I was thinking about it. The same image could suddenly look completely different even though nothing in the image had changed. The only way to confirm it was by looking at the surrounding terrain and using our experience to understand what was actually there.
It's the fitness function: Make a model which is capable of predicting the next token. The next token of what? EVERYTHING.
So what does this lead to? To a generic intelligence which is capable of responding/answering everything.
If overfitted, the model just remembers every possibility in the world but this is not possible anyway so it will start to identify patterns and rules and will use them instead.
Basically 'compressing' every possibility to every question someone could ask -> compression leads to intelligence.
> make_more_likely is, of course, doing a heroic amount of work here.
Indeed it is, and so is even just the inference method. I think it's worth remembering that both involve running the input tokens through a gargantuan neural network with (often) billions of parameters that only gain semantic meaning during the training process itself.
> it is trained to predict next tokens as they occur in its training data.
What I found important to understand is that not even the pretrainig is a deterministic process that only depends on the training data - as you would expect if the model just captured statistical properties of the data.
Gradient descent starts by setting all the parameters of the neural network to some initial values - usually by setting them at random, according to some distribution. Then during training, it gradually nudges them towards values that somehow make them useful to calculate the desired outcome of the network.
This means that by taking the exact same trainset and the exact same model architecture, you can still get models with different internal structure. The result doesn't just depend on the training data, but also on the order of examples, learning rate, the parameter initialization, etc etc.
Text renderer, whatever. Doesn't matter how you think of them, they are very interesting technology that is being misused and misconstrued in the name of something that has nothing to do with technology: political economy.
The greatest trick the rich ever pulled was making us think that the economy is about technology, and not politics.
Sure, politics and economics are involved - but why can’t technology also play an important role?
None of this AI political economy stuff was relevant in 2015 because necessary technological breakthroughs had not yet been made.
Computers have been at the center of civilization and politics since they helped win a war by changing intelligence work forever in WWII.
The idea that some new thing was born with LLMs and that this new thing fundamentally changes the calculus from the politics of labor and wealth into a technical discussion of possibilities and constraints determined by what the machine is and can do, and not what people should do, is yet another political play. It is always political, it is never moved beyond politics into technology, no matter how much technology changes.
2 replies →
I think the author is arguing against the idea of a next-token predictor as something that simply uses the weights in the neural net which record the probabilities of tokens following other tokens as a valid definition of what an LLM is. Essentially a massive extreme markov chain.
With reinforcement learning and probably attention and other tricks that affect the weights based on things that aren't strictly in the training data, so the argument goes, you can end up with tokens following strings of tokens that would not be possible to be output with the training data and original weights alone. So describing it as solely a next-token predictor is incorrect based on this framing of it.
But that's just my take on this, I'm still trying to wrap my head around it all.
Yes, I think that’s a good explanation. There are really two sides to it.
There’s the mechanical, inference time, autoregressive, one-token-after-another side, which I’m not going to argue isn’t prediction. I just think that’s a relatively uninteresting use of the word “prediction,” because it’s effectively a system predicting its own output.
The more interesting question is what happens at training time. As you describe, reinforcement learning allows the model to learn to output things that it never could have learned simply by predicting what appears in the training corpus.
More concretely, in reinforcement learning there are no ground-truth next tokens to predict.
In supervised machine learning, “prediction” usually means there is some ground-truth label that will eventually be revealed. The model predicts what that label is, the difference between the prediction and the truth gives you a loss, and you learn from that.
But in reinforcement learning, there is no ground-truth action waiting to be revealed. The model chooses an action, observes the consequences, and learns from the reward. To me, that’s a meaningfully different thing from prediction.
It literally is token prediction with vector search.
Yes there’s an app layer in the chat product for convenience and parsing but the model is exposing methods like ‘complete’ that predict out word sequences.
When model temperature settings are not added in, you get the same exact response every time, just like ngram.
What you’re seeing with Gemini and ChatGPT is context caching to prevent getting sidetracked and response boilerplates with multi-modality so they can call out to image generation, a code completion service, etc. to assemble a complete response.
That’s why the mainstream providers seem so much better (or at least consistent in replies) - each prompt is actually requesting multiple models and doing a lot of application level work to assemble the response you see. That also why they take sooo long to respond.
The language model itself is quite literally a text completion machine, with settings.
I would say that it is not even wrong. You say it’s a next token predictor. I say it isn’t. What observable behavior of the system can settle our dispute?
I fail to see how any possible output could cause either of us to change our mind.
That's perfectly put. The value of words and what we assign to them, let alone how we assign meaning, or the interpretation of what the other person says, is completely arbitrary. (in practice).
What comes after “the dog”?
Does it complete the sentence?
If not then it’s not a next token predictor. Or at least not a good one.
They are indeed "just next-token predictors". But rockets to orbit are also "just forward and upward goers". And Steph Curry is "just next three-pointer scorer" and so on. And as the article points out, chess engines are also "just next-move makers". A flawless oracular trading system is "just next trade maker". It turns out that "next n doer" is a true model that illuminates little.
The biggest problem is the word "predictor". Once you get into post training with RLHF and RLVR, it simply isn't doing that. It is not predicting anything. It's producing tokens, but it isn't predicting them. The chess analogy in the post is a good one - it's closer to searching for a set of moves that give a result than predict. It's search for a set of ideas, represented as locations in very high dimensional space, that when put together in the right order lead to a result.
> Calling the second system a "next-move predictor" would be strange.
That is EXACTLY what I would call it. I don't understand why not.
I cannot read another AI written blog post. At this point, seeing claudish language is a reliable hueristic of "not worth reading"
- It is a next token predictor OP https://blog.florianherrengt.com/how-llms-work.html
- deal with it and stop whitewashing, greywashing it
>Calling the second system a “next-move predictor” would be strange. It is not trying to predict what move appeared next in a dataset. It is trying to choose a move that wins.
Sounds like the next 1000 years depend on how carefully we define "winning".
The "next-token predictor" is taking a design stance (see Dennett). But to take the intentional stance is more interesting in case of large language/reasoning models.
[dead]
If you get down to it, any system that produces output is a "next token predictor". Every compiler, every tool that generates output. Even a web render engine is: next pixel predictor.
Perhaps a walkabout across a manifold.
The chess analogy is not great since there is a defined heuristic in chess for "winning" or "optimal board state". A system doesn't need pretraining if they can fit the rule.
I understand how a computer can know that a chess move is more likely to lead to a win, and therefore “correct”, but I don’t understand how it can know that a token is correct. Can someone explain?
The LLM produces a probability distribution over the likelihood of all possible next tokens. So whatever the tokens are, "ch", "ex", etc. the next one gets a probability.
During training, real life text is fed through the LLM, and rhe "correct" token is the one actually observed in the training text. Here's a recent video walkthrough in some detail, mostly aimed at providing a deeper understanding than "next token predictor function":
https://youtu.be/GlYgs6v2YfU?is=IxVMhoCCE4N4WRVK
(Start at 15:30 for the LLM specific parts)
Thanks - that makes sense. On that basis the article’s thesis is totally wrong - it would be like a computer program rating its ability based on how well it predicts moves played by grandmasters in the past. It’s not inventing new moves.
3 replies →
During training, certain tokens are more likely to lead to a lower loss function value, which is how you "win" the game of LLM output.
So, next-token predictors
1 reply →
>I don’t understand how it can know that a token is correct.
It can't. The next token is just the most statistically probably given the context (at least in transformers). Try a very small/weak model in your own machine and more often than not it would get stuck repeating the same word or even just output garbage. Because after training and quantization (where some information is lost), that's the most probable next token. Large models can be tricked to fall in the same behavior with very very specific inputs. Still happen, even in frontier models. And they can't detect if the output is wrong.
That's why the premise in TFA is wrong, because a transformer is a next-token predictor. It literally is that. There's nothing secret or magical, it's just a very mechanical process, with a lot of matrix multiplication, normalization, a few random passes, mappings between embeddings and a dictionary of tokens, in a very very high scale.
If someone has found something that's not a mechanical, algorithmic computation and llms are doing something nobody can explain and can't even be modeled in math, I'm happy to be educated.
classical particles obey deterministic state evolution rules, yet put 10 in a box and you cannot say where they will be 5 minutes later.
3 replies →
> It can't.
Neither can you. This is the whole debate.
It knows nothing of correctness or winning. It is predicting only what is most likely given its corpus.
My poor understanding is that an LLM does not "know" either. It basically uses probabilities to predict the next word based on a large matrix of probable outcomes.
For example, say I ask an LLM, "What sentence in English contains every letter in the alphabet?"
It would respond with something like:
"The quick fox jumps over the lazy, brown [next word]"
(Assume all the words were previously guessed correctly at this point)
The LLM guesses the last word based on what it has been trained on. Let's pretend the matrix is small, and the options narrow down to something like:
1. Dog (99.9% confidence) 2. Cow (85% confidence) 3. Bag (75% confidence) 4. Crayon (25% confidence)
The machine can confidently determine the final word of the sentence, "The quick fox jumps over the lazy, brown dog" because that sentence is unique because it is often used for testing things like fonts, a fun piece of trivia, and so on.
Brown Cow is not a bad guess because it's a type of cow and a yogurt brand. Brown bags and brown crayons are also perfectly rational adjectives to describe those common items and are not a bad guess either.
However, in the context of that sentence, dog is the most correct answer because one is unlikely to have written "The quick fox jumps over the lazy, brown crayon," thus it is quite improbable to be the answer.
My understand is this is where hallucinations can often come from. If the trivia about the sentence happened to not be in common in the data set, then "brown cow" might not be a terrible guess. There is clearly something rational behind that answer, but it's not correct in the sense that it answers the question correctly nor followed the instruction properly.
I'm sure the LLMs we have are far more capable these days. In fact, it wouldn't surprise me if an LLM could check its answer by counting the distinct letters in each word to verify. Not sure though.
Again, this is just a poor example based on my understanding, but I hope it helps (and is more correct than not).
Edit: Pretend word = token. It's technically tokens and not entire words, but I didn't not want to get into tokenization of words.
I really like the pseudo code example on this post - one of the clearest simplified explanations I've seen of how inference and training work.
Originally I had those parts written in math with probability functions and the likes (its closer to my background). Then I remembered who is my target audience... but now that I see exactly who is my target audience I'm thinking I'll should have snuck a pelican in there. All jokes aside I appreciate the comment and I'm glad that rewrite paid off!
LLMs write one word at a time, and I do too.
It is really not that complicated: words are chosen to lead somewhere.
yep "next-embedding" predictor is more correct, and not just at the end but through the layers, and folding back dimensions into that one next token is one small final step, and next-embedding could be named "next-meaning" as well, and we're getting there...
this sentence above would made a longer article if I bothered to so blog as is being blogged here
It's a next-token computer. It computes the probabilities for the next token.
Aren’t humans just “next motor action predictors”?
I like to think of LLMs as informed dice throwing
Here's my take on the "next-token predictor" idea, from a much longer article I wrote recently:
https://www.oranlooney.com/post/rose-petals/#language-models
It’s popular to dismiss LLMs as “just next token predictors.” This is technically true, but also kind of misses the point. Markov chains, RNNs, and transformers are all language models that can be described as “next token predictors,” but they don’t all work equally well. A better question to ask is: “What is this model’s inductive bias?”
A Markov chain (an -gram model) assumes the next word depends on the previous words, and that each possible combination of words has a completely independent parameter. (Andrey Markov proposed using this language model over a century ago, making it the granddaddy of modern LLMs.) So, for a vocabulary of size , there are parameters to learn. For even a smallish like 5, that already explodes the hypothesis space beyond what can be learned from even a huge text corpus like the entire internet. And, simultaneously, having a context window of only the previous 5 words is grossly inadequate for modeling real-world language. Like our FCNN above, this model suffers from having an inductive bias which is too weak.
RNNs tried to fix this problem by compressing the entire history into a single fixed-size state vector, updated one token at a time. But that compression is itself a brutal assumption: everything worth remembering about the past must survive being squeezed through a tiny bottleneck at every step. In practice, RNN models quickly lose the plot after a handful of sentences. Locally, the text they generate looks grammatically correct and meaningful, but zoom out a little and they’re basically nonsense generators. Like our naïve linear model, this model suffers from having an inductive bias which is too strong.
Transformers manage to hit a sweet spot: by keeping the recent history around as a working memory, and attending to different parts of it at different times, the transformer’s bias matches real structure in language: the referent of a pronoun, the subject of a verb, the parenthesis waiting to be closed. Not only that, but the particular structure of the transformer, basically a weighted sum of semantic vectors from the context window, has empirically been shown to somehow be a “good enough” match for the structure of real-world language found in the wild.
Transformers aren’t “smarter” than other possible language models, they just happen to land in that Goldilocks zone where their inductive bias is just right.
>Stop Thinking of LLMs as Next-Token Predictors
>Strictly speaking, the statement “LLMs are next-token predictors” isn’t wrong,
If it isn't wrong, then I will continue thinking of them as such, thank you.
Here's a much more formal definition I can come up with (which is more complete but compatible with 'next-token predictor')
LLMs are a set of functions of the type:
>typedef int Token;
>char* token2utf8(Token token)
>Token next(Token* context)
>(Token,void*) next(void* hidden_state)
Where the second next token function's runtime is O(n), and the latter is O(1). All are constant memory.
Object paradigm is more appropriate than functional definition, as the "Hidden state" coincides with private object state rather than a function paramter and return value.
>LLM.next(system_prompt) # O(len(context))
>LLM.next(user_prompt) # O(len(user_prompt)) not of system_prompt+user_prompt
That's it, that's all LLMs are, that's the interface, the rest are implementation details.
I mean. Fundamentally they are just autoregressive next token predictors. Fundamently fou can simplify to f(x) -> x+y where x is input tokens, y is the next token and f() is the model function. Yes the model function is complex but still.
It's splitting hairs. It's still a next token predictor, just not solely prior art next token predictor. It's a more and more of a desired result next token predictor.
Calling LLMs next token predictors is like calling a brain wet calculator. Technically true, but misleading as it doesn't capture the scale, the depth nor the capability.
Shrug. My intuition is LLMs predict the new word based on a tensor vector space of patterns using arithmetic and similarity scores.
What’s not intuitive to me is that through pattern matching it’s able to express logic and reasoning.
Calling an LLM a "next-token predictor" is like calling a TomTom a "next-turn predictor." It confuses the serial format of its instructions with the computation producing them, while ignoring the map, the route, the destination, and the goal -- as well as the people, businesses, traffic, and points of interest that make the map a model of an inhabited, changing world.
Better title: Continue thinking of LLMs as Next-Token Predictors
Because no, post training doesn't change that.
I'm not sure that's a useful way to think of it.
RL post-training changes the nature of what is being predicted, basically turning it from a copying machine into a goal-seeking machine.
A base model is predicting training sample continuations (copying).
A post-trained model is now steering/narrowing the base model's predictions in directions that were reinforced by RL goals.
The model is no longer predicting what the next token will be, but rather predicting what it should be in order to steer generation in the reinforced directions.
> The model is no longer predicting what the next token will be, but rather predicting what it should be in order to steer generation in the reinforced directions.
So still next-token prediction, then.
6 replies →
Sure, I get the gist of the article. I have never liked the reductionist argument that LLMs are nothing more than next-token predictors. By that rational, the human brain is really not that much different. When I am having a conversation with another person, I do not usually have every word I will respond with stored in my limited working memory. My output is often predicted based on the previous word I spoke.
> I do not usually have every word I will respond with stored in my limited working memory. My output is often predicted based on the previous word I spoke.
People don't know exactly the words that they're going to say necessarily, but tend to start with a general concept of what they're trying to communicate and only then try to put together the words (sometimes out of order). LLMs do not begin with any sort of concept they're trying to express. LLMs are simulations that attempt to reproduce what an average person might say while wired up to a huge knowledgebase.
> LLMs do not begin with any sort of concept they're trying to express.
Why do the need to? Considering they are merely tools, I actually appreciate they do not do this. A calculator can compute far better than any human, but I appreciate that calculators are not capable of expressing anything about the computations I request. I want the answer, not a conversation.
> LLMs are simulations that attempt to reproduce what an average person might say while wired up to a huge knowledgebase.
If you will allow me to be simplistic, people -- the soul, the self -- are predominately the aggregated effects of memories and experiences and the ability to retain new memories based on new experiences, no? Consider medical conditions in the dementia family of diseases. As memories fade into the ether, what remains of the self?
Also, people simulate/emulate each other all the time based on what an average, reasonable person might say. People incapable or unwilling to perform such mimicry are often labeled with all kinds of pejorative terms.
> I have never liked the reductionist argument that LLMs are nothing more than next-token predictors.
I have never heard such an argument. Recognition that LLMs are nothing more than next-token predictors does not come from reductionism. It comes from simply knowing how they work e.g. from viewing the inference code.
J.S. Bach said something similar about music and keyboard instruments.
> "There's nothing remarkable about it. All one has to do is hit the right keys at the right time and the instrument plays itself."
My issue is not with fact at face value. My issue is with how the fact is often contextually used in arguments to delegitimize and disparage LLM outputs and LLM users.
Yes, LLMs at a fundamental level are next-token predictors. But in my opinion, LLMs are very useful, imperfect next-token predictors.
There are a lot of wannabe John Henry [1] folks out there. Love LLMs or hate'em, most of those John Henry folks ain't beating these machines on a plethora of tasks.
[1] For those unaware, https://en.wikipedia.org/wiki/John_Henry_(folklore)
Once again, all models are wrong, some are useful.
I suppose this one "fails" because of "granularity?"
I'm personally getting more comfortable with the "kabillion dimensional space" one. Even before the rise of this AI thing I'd gotten comfortable with (teaching in a very generalized way) the concept of matrix/vector math as doing this sort of thing.
Start with the math required to calculate "what the building on the screen looks like" when you're playing a video game.
Then I jump to...dating websites. You give it "dimensions" like height, weight, religion, sexual preference, music preference, whether you like long walks on the beach, whatever -- then you can calculate how "close" two people are to one another.
From there, tokens and a kabillion directions.
slop slop and more slop
[flagged]