Comment by jillesvangurp
6 days ago
Neal Stephenson touches on this topic in several of his novels. Probably the most concrete of this is "Fall; or Dodge in Hell" which involves a simulation of people's scanned brains rediscovering qualia and constructing their own simulated world from scratch. In the book, two of the deceased digital "souls" eventually mate and produce digital offspring and the whole simulation starts consuming more and more resources.
His Baroque Cycle series also touches on this in several places. One funny side plot involves a freed African slave (Dappa) who speaks dozens of different languages and is highly intelligent and an aristocratic person who maintains that of course this former slave (who is obviously a lot smarter than this aristocrat) is just a trained monkey that naturally is not conscious even though he is quite clever with language. The same books also have a lot of side plots involving Leibniz and various attempts to build thinking/computing machines.
The Dappa plot is probably the closest to a lot of debates there will be around AGI with people likely to insist for all sorts of reasons rooted in philosophy, religion, etc. that even though the AGI walks, talks, etc. like a duck, it can't be a duck. At some point, we'll have AGI that pass any test we can think of and we'll still have people arguing that these cannot be conscious.
> At some point, we'll have AGI that pass any test we can think of and we'll still have people arguing that these cannot be conscious.
Is there a distinction in your mind between consciousness and intelligence? Is it possible, for example, for a machine to solve complex problems but not be conscious? Or vice versa, can an animal or a person be very unintelligent yet still conscious?
A philosophical zombie [1] is essentially a mobile, autonomous human body devoid of consciousness. Critically, still giving all of the external cues that it has in fact consciousness. It is supposed to prove that physicalism doesn't work. "The sense of consciousness" is used like a "soul, with extra steps".
In my humble opinion, which I have no way to prove or disprove, consciousness ("as a soul with extra steps") does not exist, and we are all philosophical zombies. Consciousness "as an amalgamation of complex biological signals and neural interactions that has evolved through millions of years as a successful survival strategy" does exist, and that is all that is needed.
https://en.wikipedia.org/wiki/Philosophical_zombie
I know I'm not a philosophical zombie based on
>For example, if a philosophical zombie were poked with a sharp object, it would not feel any pain, but it would react exactly the way any conscious human would.
in the Wikipedia article because I don't get on well with the sharp poking. On the basis of Occam's razor I assume all other humans and higher animals are similar. It would be odd if evolution had made me different from all the others.
3 replies →
Serves me well for answering a comment before reading the article. This is basically what the author says, even pointing to the Philosophical zombie and all. Shame on me.
The question is mainly interesting in the context of our own behavior and interactions. Because would we treat our food differently if we decided e.g. pigs were intelligent and conscious? I had some salami on my pizza yesterday. But I also believe pigs are intelligent and conscious. And they are close relatives of us genetically. Some people choose to not eat meat for this reason.
I'm of the school of thought that we are all biological computers with emergent properties like intelligence, consciousness, etc. that we might eventually succeed in replicating. Maybe at a more modest level at first. From a practical point of view, I'm more interested in intelligent agents than conscious ones. I mainly need them to do useful things for me. Too much consciousness is a double edged sword. Because then I would have to consider how my agent feels about all this. But can you really have one without the other? I don't have a good answer to this.
People have a tendency to anthropomorphize everything around them. Definitely their pets, plants, and in some cases even inanimate objects. Which doesn't help the debate because it's already happening with LLM tools. Even though they are probably still on the definitely not conscious side of the fence even when they demonstrate mildly intelligent reasoning occasionally.
At some point, people will have a hard time telling the difference with AGI. Is it all smoke and mirrors at that point or are they dealing with an intelligent/conscious thing? That's no doubt going to entertain philosophers for centuries to come. But from a practical point of view, does it really matter at that point? If we can't reliably tell the difference, is there still a difference?
> At some point, we'll have AGI that pass any test we can think of and we'll still have people arguing that these cannot be conscious.
You've already drawn your line in the sand (i.e. they are conscious). In that case, you can't also claim that we should continue producing them by the millions at the flick of a switch.
The AI-is-conscious crowd will have to choose - either they are conscious, in which case they should not be birthed, or they are not conscious in which case we can use them as tools. You can't have both and still be logically consistent.
You have strong assumptions for the underlying moral framework
I think you mean morally consistent. though even then humans don't have any real qualms about that. Dogs and livestock are conscious, we use those as tools.
> Dogs and livestock are conscious, we use those as tools.
Sure, but we don't create as many as we can, then kill them at the end of the day when the work is done.
If you want to call AIs conscious, you can't also campaign for willy-nilly creation, even if they do get a status of a working tool (dogs, etc).
If you think they are conscious, which implies laws protecting them, then the "owner" of them gets an obligation (you can't do whatever you want to a dog, for example).
1 reply →
The clinching thing for me is how the AGI is supposed to work. If it's the same as our theory of how we work, then fine, it's one of us. It wouldn't even have to work very well.
I always like to remind myself that AI is trained on material fed to it, created and written (somewhere along the line) entirely by humans - as they perceive & interpret the universe around them.
Not for long, if I have my way.
> Probably the most concrete of this is "Fall; or Dodge in Hell" which involves a simulation of people's scanned brains rediscovering qualia and constructing their own simulated world from scratch.
This is a great example for a discussion about The Hard Problem. Here is the description from the book of the inner experience of this scanned brain as it gets booted up:
> What came next could not, of course, be described without using words. But that was deceptive in a way since he no longer had words. Nor did he have memories, or coherent thoughts, or any other way to describe or think about the qualia he was experiencing. And those qualia were of miserably low quality. To the extent he was seeing, he was seeing incoherent patterns of fluctuating light. For people of a certain age, the closest descriptor for this was “static”: the sheets, waves, and bands of noise that had covered the screens of malfunctioning television sets. Static, in a sense, wasn’t real. It was simply what you got out of a system when it was unable to lock on to any strong signal—“Strong” meaning actually conveying useful, or at least understandable, information. Modern computer screens were smart enough to just shut down, or put up an error message, when the signal was lost. Old analog sets had no choice but to display something. The electron beam was forever scanning, a mindless beacon, and if you fed it nothing else it would produce a visual map of whatever was contingently banging around in its circuitry: some garbled mix of electrical noise from Mom’s vacuum cleaner, Dad’s shaver, solar flares, stray transmissions caroming off the ionosphere, and whatever happened when little feedback loops on the circuit board got out of hand. Likewise, to the extent he was hearing anything, it was just an inchoate hiss.
The Hard Problem asks: who is experiencing this qualia and why is there an experiencer at all? Stephenson writes how this simulated brain is experiencing static as it condenses into meaningful patterns, but he implicitly starts with someone experiencing this static qualia. If this is the very beginning of the simulated brain booting up, where did this experiencer come from?
> At some point, we'll have AGI that pass any test we can think of and we'll still have people arguing that these cannot be conscious.
Because you're conflating intelligence with consciousness. There is no test for consciousness. In fact, you can't even prove that other human beings are conscious, you only know that you yourself are because it's self evident to you (cogito ergo sum, I think therefore I am). The whole point of the hard problem is that you can imagine something exactly like a human being that passes every test of being a human being (e.g. an AI) but still not be sure that it has any inner experience.
>There is no test for consciousness
There's the Glasgow Coma Scale "the most widely used tool for measuring comas and decreases in consciousness"
https://my.clevelandclinic.org/health/diagnostics/24848-glas...
I know it's probably not what you're thinking of but does illustrate these things are not totally beyond testing and experimentation.