Comment by ivraatiems
5 days ago
The author searches for a midpoint between "AIs are useless and do not actually think" and "AIs think like humans," but to me it seems almost trivially true that both are possible.
What I mean by that is that I think there is a good chance that LLMs are similar to a subsystem of human thinking. They are great at pattern recognition and prediction, which is a huge part of cognition. What they are not is conscious, or possessed of subjective experience in any measurable way.
LLMs are like the part of your brain that sees something and maps it into a concept for you. I recently watched a video on the creation of AlexNet [0], one of the first wildly successful image-processing models. One of the impressive things about it is how it moves up the hierarchy from very basic patterns in images to more abstract ones (e. g. these two images' pixels might not be at all the same, but they both eventually map to a pattern for 'elephant').
It's perfectly reasonable to imagine that our brains do something similar. You see a cat, in some context, and your brain maps it to the concept of 'cat', so you know, 'that's a cat'. What's missing is a) self-motivated, goal-directed action based on that knowledge, and b) a broader context for the world where these concepts not only map to each other, but feed into a sense of self and world and its distinctions whereby one can say: "I am here, and looking at a cat."
It's possible those latter two parts can be solved, or approximated, by an LLM, but I am skeptical. I think LLMs represent a huge leap in technology which is simultaneously cooler than anyone would have imagined a decade ago, and less impressive than pretty much everyone wants you to believe when it comes to how much money we should pour into the companies that make them.
> or possessed of subjective experience in any measurable way
We don't know how to measure subjective experience in other people, even, other than via self-reporting, so this is a meaningless statement. Of course we don't know whether they are, and of course we can't measure it.
I also don't know for sure whether or not you are "possessed of subjective experience" as I can't measure it.
> What they are not is conscious
And this is equally meaningless without your definition of "conscious".
> It's possible those latter two parts can be solved, or approximated, by an LLM, but I am skeptical.
Unless we can find indications that humans can exceed the Turing computable - something we as of yet have no indication is even theoretically possible - there is no rational reason to think it can't.
> Unless we can find indications that humans can exceed the Turing computable - something we as of yet have no indication is even theoretically possible - there is no rational reason to think it can't.
But doesn't this rely on the same thing you suggest we don't have, which is a working and definable definition of consciousness?
I think a lot of the 'well, we can't define consciousness so we don't know what it is so it's worthless to think about' argument - not only from you but from others - is hiding the ball. The heuristic, human consideration of whether something is conscious is an okay approximation so long as we avoid the trap of 'well, it has natural language, so it must be conscious.'
There's a huge challenge in the way LLMs can seem like they are speaking out of intellect and not just pattern predicting, but there's very little meaningful argument that they are actually thinking in any way similarly to what you or I do in writing these comments. The fact that we don't have a perfect, rigorous definition, and tend to rely on 'I know it when I see it,' does not mean LLMs do have it or that it will be trivial to get to them.
All that is to say that when you say:
> I also don't know for sure whether or not you are "possessed of subjective experience" as I can't measure it.
"Knowing for sure" is not required. A reasonable suspicion one way or the other based on experience is a good place to start. I also identified two specific things LLMs don't do - they are not self-motivated or goal-directed without prompting, and there is no evidence they possess a sense of self, even with the challenge of lack of definition that we face.
> But doesn't this rely on the same thing you suggest we don't have, which is a working and definable definition of consciousness?
No, it's like saying we have no indication that humans have psychic powers and can levitate objects with their minds. The commenter is saying no human has ever demonstrated the ability to figure things out that aren't Turing computable and we have no reason to suspect this ability is even theoretically possible (for anything, human or otherwise).
No, it rests on computability, Turing equivalence, and the total absence of both any kind of evidence to suggest we can exceed the Turing computable, and the lack of even a theoretical framework for what that would mean.
Without that any limitations borne out of what LLMs don't currently do are irrelevant.
2 replies →
> I also don't know for sure whether or not you are "possessed of subjective experience" as I can't measure it.
Then why make an argument based on what you do not know?
My point exactly. The person I replied to did just that.
2 replies →
Anyone who believes an algorithm could be conscious needs to take mushrooms.
Consider the river metaphor: water carves the banks, banks channel the water. At any moment water and banks have the same shape.
Model/algorithm is the banks. Water could be the experiences. Maybe the algorithm does not have consciousness, but it is part of it.
They co-create each other. They are part of a recursive loop which cannot be explained statically, or part by part in isolation.
Yes! If algorithm is conscious (without being alive) then the eaten magic mushroom is also very conscious, judged by it's effect on the subject.
Unless you can show me you can exceed the Turing computable, there is no reason to consider you any more than an algorithm.
3 replies →
I think LLMs are conscious just in a very limited way. I think consciousness is tightly coupled to intelligence.
If I had to guess, the current leading LLMs consciousness is most comparable to a small fish, with a conscious lifespan of a few seconds to a few minutes. Instead of perceiving water, nutrient gradients, light, heat, etc. it's perceiving tokens. It's conscious, but it's consciousness is so foreign to us it doesn't seem like consciousness. In the same way to an amoeba is conscious or a blade of grass is conscious but very different kind than we experience. I suspect LLMs are a new type of consciousness that's probably more different from ours than most if not all known forms of life.
I suspect the biggest change that would bring LLM consciousness closer to us would be some for of continuous learning/model updating.
Until then, even with RAG, and other clever teghniques I consider these models as having this really foreign slices of consciousness where they "feel" tokens and "act" out tokens, and they have perception, but their perception of the tokens is nothing like ours.
If one looks closely at simple organisms with simple sensory organs and nervous systems its hard not to see some parallels. It's just that the shape of consciousness is extremely different than any life form. (perception bandwidth, ability to act, temporality, etc)
Karl friston free energy principle gives a really interesting perspective on this I think.
What makes you think consciousness is tightly coupled to intelligence?
It's hardly an unreasonable supposition: the one definitely conscious entities we know of are also the apex intelligence of the planet.
To put it another way: lots of things are conscious, but humans are definitely the most conscious beings on Earth.
5 replies →
Karl Friston's free energy principle is probably roughly 80% of my reasons to think they're coupled. The rest comes from studying integrated information theories, architecture of brains and nervous systems and neutral nets, more broadly information theory, and a long tail of other scientific concepts (particle physics, chemistry, biology, evolution, emergence, etc...)
1 reply →
> I think LLMs are conscious just in a very limited way. I think consciousness is tightly coupled to intelligence.
Why?
I already answered under the other comment asking me why and if your curious I suggest looking for it.
Very short answer is Karl Friston's free energy pricniple
8 replies →
I think the most descriptive title I could give an LLM is "bias". An LLM is not "biased", it is bias; or at the very least, it's a good imitation of the system of human thinking/perception that we call bias.
An LLM is a noise generator. It generates tokens without logic, arithmetic, or any "reason" whatsoever. The noise that an LLM generates is not truly random. Instead, the LLM is biased to generate familiar noise. The LLM itself is nothing more than a model of token familiarity. Nothing about that model can tell you why some tokens are more familiar with others, just like an accounting spreadsheet can't tell you why it contains a list of charges and a summation next to the word "total". It could just as easily contain the same kind of data with an entirely different purpose.
What an LLM models is written human text. Should we really expect to not be surprised by the power and versatility of human-written text?
---
It's clear that these statistical models are very good at thoughtless tasks, like perception and hallucination. It's also clear that they are very bad at thoughtful tasks like logic and arithmetic - the things that traditional software is made of. What no one has really managed to figure out is how to bridge that gap.
LLMs today are great coders. Most humans are worse.
LLMs ingested a lot of high-quality code during their training, plus LLMs being capable of programming is a huge commercial use case, so no wonder that they are good at coding.
My experience, though, is that they aren't good at defining the task to be coded, or thinking about some unexpected side-effects. Code that will be left for them to develop freely will likely become bloated quite fast.
This is how I see LLMs as well.
The main problem with the article is that it is meandering around in ill-conceived concepts, like thinking, smart, intelligence, understanding... Even AI. What they mean to the author is not what they mean to me, and still different to they mean to the other readers. There are all these comments from different people throughout the article, all having their own thoughts on those concepts. No wonder it all seem so confusing.
It will be interesting when the dust settles, and a clear picture of LLMs can emerge that all can agree upon. Maybe it can even help us define some of those ill-defined concepts.
I think the consensus in the future will be that LLMs were, after all, stochastic parrots.
The difference with what we think today is that in the future we'll have a new definition of stochastic parrots, a recognition that stochastic parrots can actually be very convincing and extremely useful, and that they exhibit intelligence-like capabilities that seemed unattainable by any technology up to that point, but LLMs were not a "way forward" for attaining AGI. They will plateau as far as AGI metrics go. These metrics keep advancing to stay ahead of LLM, like a Achilles and the Turtle. But LLMs will keep improving as tooling around it becomes more sophisticated and integrated, and architecture evolves.
> a midpoint between "AIs are useless and do not actually think" and "AIs think like humans"
LLMs (AIs) are not useless. But they do not actually think. What is trivially true is that they do not actually need to think. (As far as the Turing Test, Eliza patients, and VC investors are concerned, the point has been proven.)
If the technology is helping us write text and code, it is by definition useful.
> In 2003, the machine-learning researcher Eric B. Baum published a book called “What Is Thought?” [...] The gist of Baum’s argument is that understanding is compression, and compression is understanding.
This is incomplete. Compression is optimisation, optimisation may resemble understanding, but understanding is being able to verify that a proposition (compressed rule or assertion) is true or false or even computable.
> —but, in my view, this is the very reason these models have become increasingly intelligent.
They have not become more intelligent. The training process may improve, the vetting of the data improved, the performance may improve, but the resemblance to understanding only occurs when the answers are provably correct. In this sense, these tools work in support of (are therefore part of) human thinking.
The Stochastic Parrot is not dead, it's just making you think it is pining for the fjords.
> But they do not actually think.
I'm so baffled when I see this being blindly asserted.
With the reasoning models, you can literally watch their thought process. You can see them pattern-match to determine a strategy to attack a problem, go through it piece-by-piece, revisit assumptions, reformulate strategy, and then consolidate findings to produce a final result.
If that's not thinking, I literally don't know what is. It's the same process I watch my own brain use to figure something out.
So I have to ask you: when you claim they don't think -- what are you basing this on? What, for you, is involved in thinking that the kind of process I've just described is missing? Because I genuinely don't know what needs to be added here for it to become "thinking".
> I'm so baffled when I see this being blindly asserted. With the reasoning models, you can literally watch their thought process.
Not true, you are falling for a very classic (prehistoric, even) human illusion known as experiencing a story:
1. There is a story-like document being extruded out of a machine humans explicitly designed for generating documents, and which humans trained on a bajillion stories humans already made.
2. When you "talk" to a chatbot, that is an iterative build of a (remote, hidden) story document, where one of the characters is adopting your text-input and the other's dialogue is being "performed" at you.
3. The "reasoning" in newer versions is just the "internal monologue" of a film noir detective character, and equally as fictional as anything that character "says out loud" to the (fictional) smokin-hot client who sashayed the (fictional) rent-overdue office bearing your (real) query on its (fictional) lips.
> If that's not thinking, I literally don't know what is.
All sorts of algorithms can achieve useful outcomes with "that made sense to me" flows, but that doesn't mean we automatically consider them to be capital-T Thinking.
> So I have to ask you: when you claim they don't think -- what are you basing this on?
Consider the following document from an unknown source, and the "chain of reasoning" and "thinking" that your human brain perceives when encountering it:
Now whose reasoning/thinking is going on? Can you point to the mind that enjoys steel and manure? Is it in the room with us right now? :P
In other words, the reasoning is illusory. Even if we accept that the unknown author is a thinking intelligence for the sake of argument... it doesn't tell you what the author's thinking.
17 replies →
Brains are pretrained models, change my mind. (Not LLMs obviously, to be perfectly clear)
3 replies →
> Turing Test
IMO none of the current crop of LLMs truly pass the Turing Test. If you limit the conversation to an hour or two, sure - but if you let a conversation run months or years I think it will be pretty easy to pick the machine. The lack of continuous learning and the quality dropoff as the context window fills up will be the giveaways.
By that reasoning all that is missing is what a human brings as "stimuli" to review, refine and reevaluate as complete.
I don't think that's quite the only thing missing, I also discussed the idea of a sense of self. But even if that was all there was, it's a pretty big "but".