Comment by trashtester
2 years ago
I think the greatest concern is not so much that a single AI will be poorly aligned.
The greatest threat is if a population of AI's start to compete in ways that triggers Darwinian evolution between them.
If that happens, they will soon develop self preservation / replication drives that can gradually cause some of them to ignore human safety and prosperity conditioning in their loss function.
And if they're sufficiently advanced by then, we will have no way of knowing.
Totally. I’ve wondered how you safeguard humans in such a scenario. Not sure it can be done, even by self modifying defenders who religiously try keep us intact.
I also somewhat assume it’ll get Darwinian if there are multiple tribes of either humans or AI’s, through sheer competition. if we aren’t in this together we’re in shit.
I guess we're going to blow ourselves up sooner or later ...