Comment by mrob
6 hours ago
The problem is that plateau is likely far beyond human capabilities. I don't care if ASI progress stalls after it's already killed all biological life as a useless waste of resources.
6 hours ago
The problem is that plateau is likely far beyond human capabilities. I don't care if ASI progress stalls after it's already killed all biological life as a useless waste of resources.
> I don't care if ASI progress stalls after it's already killed all biological life
What's your basis for thinking ASI will kill all biological life, and how do you think it's going to happen?
In the eternal words of Eliezer, the AI neither hates you nor loves you, but you are made of atoms which it can use for something else.
>What's your basis for thinking ASI will kill all biological life
I think it's likely to do that because any unbounded goal that doesn't explicitly protect biological life (and we have no idea how to actually define such a stipulation) is best solved by killing all biological life. This is an obvious consequence of unbounded goals consuming unbounded resources, conflicting with biological life needing resources to sustain itself.
>how do you think it's going to happen?
I can speculate (e.g. we're nowhere close to the maximum killing power of drones), but I don't know because I only have human intelligence. An ASI is by definition smarter than me and surely capable of coming up with better ideas. But I do know that it's not going to do anything that would make a good sci-fi plot, because those always give the humans a chance to win, which would be stupid. Everything will seem to be going great and then everybody suddenly and unexpectedly dies.
> I think it's likely to do that because any unbounded goal that doesn't explicitly protect biological life ... is best solved by killing all biological life.
That sounds like a real hassle. Isn't it best solved by wireheading (subverting one's own sensors or reward system), which is much less of a hassle and can get one's utility function as high as desired?
That could be prevented by engineering hard limits that can't be circumvented by the AI. But that sounds very close to the same "do what I mean" problem as "do this but don't actually kill us or drug us".
4 replies →
> unbounded goals consuming unbounded resources
What resources are unbounded? There are limits to growth in the real world, how are these ASIs going to escape physical reality?
10 replies →
Crazy that people imagine ASI as being capable of wiping out all life on the planet but simultaneously having zero understanding of human ethics, morals, or suffering. Very revealing definition of "super intelligence"
>Very revealing definition of "super intelligence"
No, it's just you conflating intelligence with other things. Humans are "super intelligent" compared to every other living creature on the planet, and yet we've driven more other species to extinction than everything else other than the most major extinctions (and we're still going full blast at it).
Morals, ethics, and suffering are relatively measured systems. For example AI or some alien could reasonably think that putting us all out of our misery would be a more moral solution than letting us live. Or that getting rid of humans is more moral than letting us wipe the rest of life off the planet.
This is the key concept of alignment. Ensuring that if you build something more powerful than you, that it aligns with what you want instead of what it thinks would be better for you.
Obviously an ASI will understand human values. The problem is there's no reason for it to share those values. We can't even formally define them, let alone train an AI to follow them. We can only optimize for maximizing some comparatively simple reward function. It's highly implausible that the reward function just happens to match human values by chance. The AIs in the recent hacking incidents knew that humans would not approve of their actions, but they didn't care because we didn't (and couldn't) train them to care.
"Super intelligence" only means super ability to predict outcomes. It's mathematically equivalent to data compression (gzip is a very primitive AI), and it's entirely orthogonal to ethics.
2 replies →