Comment by hkalbasi
3 days ago
> A LLM (...) merely produces syntactically likely combinations of the training data
While doesn't matter much in the rest of the policy, this is a common misconception among AI skeptics. It is not the case for a long time (since RL is used heavily in the training) and a LLM may go beyond its training data.
I think the questionable term is not “syntactically likely” but “merely”. Syntactical likeness is a vast solution space that encompasses the work of a terrible developer and a genius developer. In fact this solution space is a gap wide enough to encompass all coding knowledge and expertise.
"Combinations of the training data" is also hiding a pretty common and serious misconception about modern generative AI: this seems to come from a mental model that LLMs and image generators are merely producing a collage of sorts, especially that this is some way linked to specific parts of the training data as opposed to a generalized result of building up an understanding of how to produce a sentence, piece of code, or plausible image.
That is how they feel when using them for code. It feels like they generate templates filled in with the details of a specific case.
The natural next argument that I see a lot is "well it's still all probabilistic", which is technically true. However, don't the atoms that make up the cells that make up a human move around and interact based on probabilities?
Saying a LLM is all just based on probabilities is pretty meaningless, even if it is true, since everything else is just based on probabilities too. What matters is the massive amount of machinery that generates those probabilities. Anything from rolling a dice to an accurate model of an entire human, or even a model of the entire universe, is all "just probabilities".
That argument is mostly used against the misconception of AI-stans that LLMs are compilers, English is a programming language, and that prompting is a programing. There are many reasons for why that is false, and LLMs being stochastic is one of the more obvious reasons.
But there are more probabilistic elements about LLMs. Inference is by default deterministic for any given input and weights, however it is made stochastic algorithmically, when the model chooses the second or third likeliest outcome with some probability. Training is stochastic, the adjustment of the weights is probabilistic, etc. etc.
You seem to make the fact that our brains are also probabilistic as some sort of a gotcha. However that has never been in dispute. Neurons fire with certain probability, this has been known since the advent of neuroscience. However this is why computers are preferred for certain tasks over human brains. When you run a program you can be certain it behaves in a certain way given a particular input and parameters.
The "gotcha" here, if you want to call it that, is that the picking the next token based on probabilities (either in training or in the sampler) is not an argument that LLMs are inherently flawed, especially when one reduces a LLM to "it just picks a token based on a probability" as you could apply that same description to a human.
1 reply →
I assume RL stands for reinforcement learning, which just means that the weights are adjusted by evaluating some loss function. If the loss function uses the training data (i.e. supervised learning) then the original assumption still holds. So I fail to see how this is a misconception.
I think OP might be referring to synthetic data, which is now used to provide more data for training?
That would makes sense. This is how AlphaGo became better then the training data, and was the basis for Alpha Zero which used no human data and became even better.
However the loss function (or the success criteria more broadly) for the game of go is extremely simple. We do not know whether such a success criteria even exists for a generic task like coding, and it is a mistake to assume that the AI labs have found one.
Supervised learning and reinforcement learning are mathematically distinct. Supervised pre-training as you said maximizes the likelihood of a token sequence. RL optimizes a policy against an external reward signal or execution environment. For example, RL evaluates code against runtime execution/unit tests/proof checkers (e.g lean). Models learn new strategies and are able to produce novel code if trained in an RL environment.
I don‘t understand what you mean. Supervised learning is reinforcement learning, while not all reinforcement learning is supervised learning (e.g unsupervised learning is also reinforcement learning; i.e. reinforcement learning with unlabelled data).
Like you said, you can have reinforcement learning which doesn’t use training data. But that is not what my parent said. What they said is: since RL is used heavily in the training. And since reinforcement learning is a broad category which includes supervised learning, nothing in their logic disproves the strawman they created from an AI skeptic.
1 reply →
Doesn't the LLM still output the next most probable token (modulus heat/desired variation) according to its model? Training shifts the probability - but doesn't change the fact that the output is a sampling based on input and the model?
So do humans more or less? The only counterpoints I've heard are religious or unmeasured quantum brain something.
I think there are quite fundamental differences to the LLM model equivalent to an adult human mind and current LLMs (even accepting an LLM can model a human brain, which I don't think is a given - but I also don't think it's impossible).
The details matter (training set, full context). I doubt our current LLMs knows what smelling cut grass on a wet morning feels like, or how the stomach flutters on a first kiss - even if they have "read" tens to hundreds of attempts at describing such things.
You're essentially suggesting we play Turing's Guessing Game; if your LLM can guess my response to any prompt - then it can be said to have modeled how I would reply to those prompts. Expand the prompts far enough, and you could reasonably argue the LLM models me.
2 replies →
No. Humans actually understand things and can reason from one fact to another. We do not just spit out the next most likely thing without any intelligence the way an LLM does.
The combination of the most probable tokens doesn't necessarily have to be in the training set, thus creating something completely novel.
If you shuffle a deck of cards, you likely have a novel sequence of cards. I'm not sure how that is relevant?
1 reply →
That is called a hallucination.
5 replies →
It’s not sampling anything though. At each inference step the fundamental output of the process can be interpreted as a probability for each possible token how likely it is. Sampling would imply otherwise but the probabilities are the output.
I would like to offer this comment from three days ago as counterpoint:
https://news.ycombinator.com/item?id=49013694
Common sense testing only tells about the presence of common sense, that's it. It is clear from recent events that LLMs do definitely possess some sort of intelligence even though it might have been polarized to certain domains.
What does that prove? That AIs are not RLd for counting letters in texts? Do you doubt that they could excel at it with only synthetic data?