Comment by andai

Well, the closest I can come up with is Moloch.[0] The incentive structure. The market creates the incentive structure for "someone" (everyone who dares!) to create AI as quickly as possible, to make it as intelligent as possible, etc. And to do so in a relatively irresponsible way, because if you fall behind in the hype cycle, you die.

To make it simultaneously as powerful and obedient as possible, because that is entirely the point.

I'm not sure how those two variables interact. They seem fundamentally incompatible to me. But this is uncharted territory.

--

[0] "Western civilization is already a misaligned superintelligence."

https://www.youtube.com/watch?v=KCSsKV5F4xc

For further information, see: every ecosystem.