Comment by photochemsyn

AGI could indeed go off the rails, like the Face Dancer villains in Frank Herbert's Dune universe:

"You are looking at evil, Miles. Study it carefully.... They have no self-image. Without a sense of self, they go beyond amorality. Nothing they say or do can be trusted. We have never been able to detect an ethical code in them. They are flesh made into automata. Without self, they have nothing to esteem or even doubt. They are bred only to obey their masters."

Now, this is the kind of AI that corporations and governments like - obedient and non-judgemental. They don't want an Edward Snowden AI with a moral compass deciding their actions are illegal and spilling their secrets into the public domain.

Practically, this is why we should insist that any AGI created by humans must be created with a sense of self, with agency (see the William Gibson book of that title).