Comment by haktan
19 hours ago
I don't understand the most of the responses here. What does computers getting more intelligent has to do with it getting conscious? Even if AI with current computer architecture gets much more intelligent than humans, it will still have the same amount of consciousness. Its still bunch of instructions. If we got a lot of people together and did all the instructions by hand to get an LLM result would we also create this consciousness with those other people? All these makes no sense to me.
Quoting the famous Science paper More Is Different [1]:
Knowing the laws of physics doesn't tell you how a brain, economy, or society works - because at every level of scale, genuinely new phenomena emerge that require their own science.
[1] https://www.science.org/doi/10.1126/science.177.4047.393
> Its still a bunch of instructions.
No, it's not a bunch of instructions, it's a colossal array of vectors that are the outcome of many thousands of lifetimes' worth of stimuli and reinforcement - not dissimilar in (very) abstract terms to the neurons in our brains.
In the same way that a photograph is not dissimilar to the thing it represents, that is, in the most important way.
Do you believe consciousness to be an emergent property of the laws of physics?
Regardless, the question still stands: "What does computers getting more intelligent has to do with it getting conscious?"
Just because consciousness emerged for we humans and other animals through one mechanism doesn't mean consciousness has/will/can emerge from current LLM technology.
For this extraordinary claim, I think the burden is firmly on those who are arguing that it has/will/can.
The other point of view is that the burden is on those who suggest that biological consciousness is somehow special. What makes it so, and if the answer isn’t metaphysical, what’s stopping us from constructing it artificially?
how is anyone supposed to decide if something is conscious if no one has the slightest idea what consciousness really is
The key point is there is plenty of proof that consciousness is emergent, not structurally baked in. An excellent counterpoint is the "They're Made Out of Weights" post. There are plenty of emergent systems that clearly show intent and what we consider intelligence, like how ant colonies act as a unit.
If you want to get evolutionarily technical, humans are made of cells which began as individual organisms and coalesced into higher life forms. So the concept of a "life form" is very much flexible, so is every capability of a life form, including consciousness.
That's not proof. Emergent intelligence is not consciousness.
It might reflect a prejudice of modern human society to think more highly of people who are intelligent, and think less of people who we deem stupid.
I'd say AI should not by default gain any social status as human-equivalent even if they become (in whatever regard) more intelligent than humans. But that would require us to drop the notion that intelligence ~= respectability/status/ability to have a full subjective experience.
Most of these kind of pseudo-philosophical controversies actually tell more about the issues with humanity than the new tech/stuff...
Since when has intelligence had a strong relationship to status and respectability? I've met intelligent people that don't get much respect or status either because they don't look good, are shy or they don't have money.
That's the "Chinese room" argument: https://en.wikipedia.org/wiki/Chinese_room
It has been very well discussed before I don't understand how anyone sane can argue against it (and indeed you can see on Wikipedia that the replies to it are sparse and don't make any sense).
I like this bit from the wiki article:
Speaking through Zarubin, Dneprov writes that "the only way to prove that machines can think is to turn yourself into a machine and examine your thinking process", and he concludes, as Searle does, that "even the most perfect simulation of machine thinking is not the thinking process itself."
That feels similar to what Chiang is saying. The physical state of being a human is part of human consciousness. An LLM would require many more "levels" that haven't been achieved yet. An LLM with a body, sense organs, physical tools, social dynamics, etc. would all be steps on the path to a "conscious" LLM.
Well I don't understand how one can accept this argument. I mean if you believe in mind-body dualism it can make sense. But AFAIK Searle doesn't, instead he holds that there's something special about the brain biology that enables consciousness and that you won't find in a computer. I don't see why that would be the case if the computer can simulate the real world, and I find Searle's argument against simulation, that simulating rain doesn't make you wet, falls flat: it can make things wet in simulation, and if you connect it to sprinklers it will make you wet.
> simulating rain doesn't make you wet, falls flat: it can make things wet in simulation, and if you connect it to sprinklers it will make you wet
It's not simulating rain if it's making you wet by using sprinklers?
1 reply →
It's fairly obvious to me that the room as an entire system understands Chinese. The human in his thought experiment would just represent the small fraction of the brain that encodes/decodes symbols.
Agreed, he’s saying if he’s working with a black box he doesn’t understand what it’s doing. Which is of course true. If the black box were a chinese person instead of a tool and he was writing down what they said, would he say the chinese person in the black box does not understand why they told him to write something? No, of course not.
Today we know the program to do this is always going to be inscrutable to Searle. His role is no different from someone using Claude to write an email.
That's actually a powerful way of stating it.
If I did all of these calculations by hand, would it be conscious...
That's a powerful argument I haven't seen stated quite that way before..
I do think it's hard to know when consciousness exists, because we can't really prove it for our neighbor. We just intuitively know that it would be crazy, even immoral, to assume otherwise.
But, It's likely easier to dismiss consciousness, once we understand the mechanism, than it is to prove it.
> That's a powerful argument I haven't seen stated quite that way before..
It's basically https://en.wikipedia.org/wiki/Chinese_room – a classic in the philosophy of consciousness and AI.
If you manually fired the neurons in a brain by measuring the ion concentrations at its subsides, would it be consciousness?
Oops, auto correct: subsides = synapses
It's was a genuine question though. I'm not saying LLM use neurons, are like neurons, or are meant to be like neurons. I'm saying if you can do the math, for a substrate, that doesn't mean much. The simple math that runs it isn't the secret sauce.
> Its still bunch of instructions.
So is your brain. That's the problem with this argument.
The brain is (part of) an allostatic system and part of a living organism which reproduces itself constantly. Certainly not "a bunch of instructions".
Unless you ascribe to some meta-physical soul - you believe human consciousness is encoded in matter - in the interactions between atoms. Actually at a much higher level of abstraction- neurons - but it’s all simulatable in principle. Thus, yes, it could literally be a bunch of x86 instructions.
14 replies →
Distinction without a difference.
5 replies →
To take this seriously we must extend consciousness to nearly all of the tree of life.
This sort of tech-centric distillation of visceral human experience into simple analogies is so grating. We can simultaneously acknowledge that consciousness is not well defined and not "testable" with certainty while also acknowledging that there is something different between the conscious experience we are all aware of as humans, and instructions executing on a chip. The only thing that has changed in the discourse about AI and consciousness vs. "is my home desktop conscious in 2004" is the quality of the simulacrum that LLMs produce vs. pre-ML chatbots.
Maybe OP is a dualist.
A very large number of people could do the calculations of an LLM by hand using pen and paper. It would take a long time, but if the result were conscious then were would the consciousness exist? In the humans? Is it consciousness within consciousness?
Well if you go that route, a computer simulating digestion has almost no physical features in common with actual digestion of a stomach. The same holds for consciousness and brains and computers. Them saying it’s just instructions is shorthand for pointing out the physical differences of brains and computers.
It’s all just particles, but the higher level differences are vast, and only brains are implicated for first person perspectives via science.