Comment by layer8
5 days ago
Sometimes after a night’s sleep, we wake up with an insight on a topic or a solution to a problem we encountered the day before. Did we “think” in our sleep to come up with the insight or solution? For all we know, it’s an unconscious process. Would we call it “thinking”?
The term “thinking” is rather ill-defined, too bound to how we perceive our own wakeful thinking.
When conversing with LLMs, I never get the feeling that they have a solid grasp on the conversation. When you dig into topics, there is always a little too much vagueness, a slight but clear lack of coherence, continuity and awareness, a prevalence of cookie-cutter verbiage. It feels like a mind that isn’t fully “there” — and maybe not at all.
I would agree that LLMs reason (well, the reasoning models). But “thinking”? I don’t know. There is something missing.
> Sometimes after a night’s sleep, we wake up with an insight on a topic or a solution to a problem we encountered the day before.
The current crop of models do not "sleep" in any way. The associated limitations on long term task adaptation are obvious barriers to their general utility.
> When conversing with LLMs, I never get the feeling that they have a solid grasp on the conversation. When you dig into topics, there is always a little too much vagueness, a slight but clear lack of coherence, continuity and awareness, a prevalence of cookie-cutter verbiage. It feels like a mind that isn’t fully “there” — and maybe not at all.
One of the key functions of REM sleep seems to be the ability to generalize concepts and make connections between "distant" ideas in latent space [1].
I would argue that the current crop of LLMs are overfit on recall ability, particularly on their training corpus. The inherent trade-off is that they are underfit on "conceptual" intelligence. The ability to make connections between these ideas.
As a result, you often get "thinking shaped objects", to paraphrase Janelle Shane [2]. It does feel like the primordial ooze of intelligence, but it is clear we still have several transformer-shaped breakthroughs before actual (human comparable) intelligence.
1. https://en.wikipedia.org/wiki/Why_We_Sleep 2. https://www.aiweirdness.com/
There is simply put no ongoing process and no feedback loop. The model does not learn. The cognition ends when the inference cycle ends. It's not thinking, it just produces output that looks similar to the output of thinking. But the process by which it does that is wholly unreleated.
Humans with certain amnestic syndromes are incapable of learning. That doesn't make them unintelligent or incapable of thought.
Most AI tooling is shipped with a feedback loop around the LLM. The quality of Claude Code for example lies in the feedback loop it provides on your code. Maybe the LLM itself isn't thinking, but the Agent which ships an LLM plus feedback loop definitely shows thinking qualities.
Just now in an debugging session with claude code:
How is this not thinking?
without getting into theory of mind it's a bit difficult to elaborate, and I don't have the time or the will for that. But the short version is that thinking is interconnected with BEING as well as will, and the Agent has neither, in a philosophically formal sense. The agent is deterministically bound. So it is a fancy Rube Goldberg machine that outputs letters in a way that creates the impression of thought, but it is not thought, in the same way that some birds can mimic human speech without even the slightest hint as to the words' or sentences' meaning, underlying grammar, connotations, subtext, context, intended use, likely effect, etc. Is speech speech if the speaker has no concept whatsoever of said speech's content, and can not use it to actualize itself? I'd say no. It's mimicry, but not speech. So that means speech is something more than just its outward aspect - the words. It is the relation of something invisible, some inner experience known only to the speaker, VIA the words.
Whereas a gorilla who learns sign language to communicate and use that communication to achieve aims which have direct correlation with its sense of self - that's thought in the Cogito, Ergo Sum sense of the word.
Thought as commonly concieved by the layman is a sort of isolated phenomenon that is mechanical in nature and can be judged by its outward effects; whereas in the philosophical tradition defining thought is known to be one of the hard questions for its mysterious qualia of being interconnected with will and being as described above.
Guess I gave you the long answer. (though, really, it could be much longer than this.) The Turing Test touches on this distinction between the appearance of thought and actual thought.
The question goes all the way down to metaphysics; some (such as myself) would say that one must be able to define awareness (what some call consciousness - though I think that term is too loaded) before you can define thought. In fact that is at the heart of the western philosophical tradition; and the jury consensus remains elusive after all these thousands of years.
7 replies →
You're moving the goalposts and contradicting yourself with language games.
Something doesn't need to learn to think. I think all the time without learning.
There's also an argument for machines already starting to crack learning with literal reinforcement training and feedback loops.
Your language game was when you said the 'cognition ends...', as cognition is just a synonym for thinking. "The thinking ends when the inference cycle ends. It's not thinking'" becomes a clear contradiction.
As for "the process by which it does that is wholly unrelated", buddy it's modelled on human neuron behaviour. That's how we've had this generative AI breakthrough. We've replicated human mental cognition as closely as we can with current technology and the output bears striking resemblance to our own generative capabilities (thoughts).
Happy to admit it's not identical, but it's damn well inside the definition of thinking, may also cover learning. It may be better to take a second look at human thinking and wonder if it's as cryptic and deep as we thought ten, twenty years ago.
Interesting, you think the associations your brain comes up with during sleep are NOT thinking?
I’d say it’s more like annealing. Sort, filter, compress, integrate.
Updates your models for the next morning, which is why the answer is there when it wasn’t before.
I assume most people agree with that framing. “Thinking” generally implies a level of consciousness or self-awareness.
“Let me think about this.” “I have to think on it.”
My brain regulates all sorts of processes unconsciously, like breathing, for example. I don’t treat those as “thinking,” so I don’t know why other unconscious brain activity would be either.
I don't know that most people think about that at all. For me, I have learned not to dismiss the remuninating, reordering, reframing that my brain performs without me deliberately reasoning line by line about it. Limiting my brain to deliberate line by line reasoning would be very limiting and costly.
"Thinking" to me is very much NOT just conscious reasoning. So much of what I think is not done consciously.
Indeed "let me think about it" is often simply giving my brain time to "sit on it", for another expression - only after which will I have enough mind time on the various alternatives for a worthwhile conscious decision.
Perhaps this is an artefact of instantiation - when you talk with an LLM, the responding instance is just that - it comes into being, inhales your entire chat history, and then continues like the last chap, finishes its response, and dies.
The continuity is currently an illusion.
Do LLMs ever ask for you to clarify something you said in a way a person who doesn't quite understand what you said will do?
Sometimes I think people leveraging criticisms of LLMs used ChatGPT 3 years ago and haven't touched one since, except for asking how many r's are in strawberry a year and a half ago.
Yeah, as someone who has gained a lot of interaction skills by playing with the constructivist learning ennvironment called the enneagram, I can attest that it much resembles behaviour characteristic of certain enneatypes.
Yes, often
What now, two minutes using one and you are going to get that!
> When conversing with LLMs, I never get the feeling that they have a solid grasp on the conversation. When you dig into topics, there is always a little too much vagueness, a slight but clear lack of coherence, continuity and awareness, a prevalence of cookie-cutter verbiage. It feels like a mind that isn’t fully “there” — and maybe not at all.
Much like speaking to a less experienced colleague, no?
They say things that contain the right ideas, but arrange it unconvincingly. Still useful to have though.
> Would we call it “thinking”?
Yes I would.