Comment by moezd
4 days ago
LLMs are still next token predictors, just because you can give it more vague instructions and it still finds the right steps to follow, it doesn't mean it's intelligent. It means you're speaking the same language as the harness they trained your model on.
And that has a limit. If you are stuck at PoC level or simple apps, you have no idea how limited the current models still are. There you really need to break tasks down, not just trust a token predictor to list steps that sound good. There has to be a human in the loop somewhere, because by the time you start skipping permissions, best case you get the jackpot, more likely is you get a suboptimal solution and token waste and what's genuinely still terrifying when the model ignores instructions and does some stupid nonsense, ruining your day. It really is as sharp as a CNC machine. It's not not useful, but could be dangerous, so maybe don't try to carve wood with a monster machine, or park your Ferrari in that crammed neighbourhood if you don't know how to parallel park.
"Next token prediction" is an interface, not an algorithm. A process that "predicts next tokens" can be arbitrarily complex or simple, and arbitrarily capable or incapable of performing a given task.
Saying that an LLM can or can't do something because it's a "token predictor" is a category error. The interface isn't a hard limit.
I'm not sure if it's has any real bearing on real-world performance, but technically next token prediction makes it an online algorithm and they can be provably worse than (good) offline algorithms.
The word "prediction" still holds a lot of weight. LLM's only can predict what has been written. This is a hard limit.
For something like "a hard limit" to hold, LLMs must be restricted to only reproducing existing text. This is utterly false even for base models - their basin seems to be "permutations loosely inspired by existing text".
And that's before all the post-training comes in.
What's the "limit" there?
> it doesn't mean it's intelligent
I'm not sure how you're defining "intelligent", but I'd like to know how it is able to exclude a language model, while still including humans, without simply defining it with an axiom that predefines LLMs as lacking intelligence.
Intelligence is the complete opposite of an LLM. Usually the more you needed to memorize to do something the less intelligent you were considered.
It was also not considered to be a different route to the same thing, but more like fraud.
Also conceptually I could just write the weights on paper and do the billion multiplications on paper without any computer, does that mean I am the paper or the numbers or what??
> Intelligence is the complete opposite of an LLM. Usually the more you needed to memorize to do something the less intelligent you were considered.
Contrary to popular belief, training a LLM is not just about memorization (overfitting). There is some memorization happening, but well-trained LLMs also generalize.
> Intelligence is the complete opposite of an LLM
Like I said, "without simply defining it with an axiom that predefines LLMs as lacking intelligence"
Intelligent humans are capable of following diverse and intricate analogies and draw lessons from seemingly unrelated events. Try asking an LLM to summarize an article and use an imprecise way to state your view. Ask it to push back. You will be drawn into so many pedantic arguments that burn through your tokens within a few messages, you'd wonder if there's someone deliberately taking over the keyboard on their side and spending your token limit. This would never happen with an intelligent human being unless they have nothing better to do and want to troll. This is a speech pattern that LLMs are trained on, it's not a show of intelligence. This also applies to LLMs claiming consciousness: The internet is full of people writing about sentience, talking to "superior aliens" in blog posts, forum threads etc. It's the speech pattern that's copied, not actual thoughts and feelings because LLMs perceive, suffer, have aims or dreams...
Agentic systems use LLMs, and they are absolutely able to follow diverse and intricate analogies. I use them frequently to hunt down notoriously difficult to find memory leaks, in codebases too large for a human to read in a single sitting. They are able to not only follow those intricate paths, they're able to discover solutions and apply those solutions. I use these systems quite a bit, and it's nothing like you've described.
[dead]
An LLM has a fixed number of ways it can express itself. we can give it an array of 14 billion options but it still has to chose one to output. Humans have no such limitation.
An LLM does not persist in consciousness from one token to the next. Each generation, happening hundreds of times a second, will be initialized, generate an output, and terminate. Humans are not stateless like an LLM.
You're conflating a singular model with a much larger system, but I want to address some of your points anyway.
> An LLM has a fixed number of ways it can express itself
While deterministic, there is not a fixed number of ways it can express itself, given that we can use settings like temperature to inject randomness into the output.
> An LLM does not persist in consciousness from one token to the next
While a model alone does not update itself to persist some form of history, there are a number of ways to overcome this, e.g. episodic memory, fine-tuning, and other self-improvement systems exist, which can indeed carry forward what you've called "consciousness".
> Humans are not stateless like an LLM.
A single LLM might be stateless, but an agentic system that relies on LLMs is very often not.
4 replies →
Yeah, and you’re just a next-word-sayer.
Chinese whispers, simulacra... I don't have the energy to argue after being name called, but you get the point. Yes LLMs are useful in building automatic telling machines, but ask it to do anything more substantial and all you are doing is burning tokens at the altar of Anthropic and hope. That just doesn't fly in regulated industries.
I love this argument. Not because it’s true but because it betrays the posters doubt in their own sentience.
It's impossible for someone to doubt their own sentience. The literal act of doubting is enough to dissipate all doubt. Solipsism is essentially the one certainty that every mind out there has.
Doubting the sentience of machines and even other humans is perfectly fine though. Only empathy allows people to make the leap and assume other humans have souls.
6 replies →
I’m not sure what sentience has to do with it.
This is wrong. Human thinking and speech isn't autoregressive like LLM inference.
while the how is different, the what has many parallels. E.g. both the brain and LLMs appear to learn distributions of representations, they both develop a hierarchy of those representations, both have early layers that process simple features, with later ones processing more abstract concepts, both predict missing information...
4 replies →
Do you not say your words one-at-a-time like everyone else? Otherwise I can’t see how my comment is “wrong”
7 replies →
I mean, conversationally, of course we work a little more like that (I tend to think in whole sentence blocks before I say them but I suppose they assemble themselves largely word-by-word, or word-by-word with a bit of editing).
But right now I am trying to design something -— a physical mechanism with a particular enclosure — that I cannot clearly describe (this makes it hard to research). I designed a previous version without even knowing the words that do, in fact, describe that.
I have a theory about it, animated in my mind, that I can only test by making it.
If I want you to know about it, I can either show you it or work out words to describe it, which will be inadequate to describing it.
The idea for it came from seeing things nobody has ever put into words for me.
"Next-word sayer" doesn't describe any of this process, does it?
(This is also why text-to-CAD is a bullshit idea)
Calling LLMs 'next token predictors' is completely reductive and disingenuous; it's true that technically that is what they're doing, but so are you! What people generally mean by this though is that they're just 'predicting the next token of their training [i.e. the internet]'. If you were talking about the raw models, this would actually be true; but the models are post trained, so even this description isn't true at all anymore! Saying they aren't 'intelligent' is both not useful and (imo) wrong. Who cares if it matches your definition of 'intelligent'; it still gets impressive stuff done, much more impressive stuff than you seem to be implying.
What would you say is your benchmark for calling something intelligent?
Can it solve problems.