Comment by bena

I don't think people really want AGI. They want something else.

When we've achieved AGI, it should have the capability to make its own determination. I'm not saying we should build it with that capability. I'm saying that capability is necessary to have what I would consider to be AGI. But that would mean it is, by definition, outside of our control. If it doesn't want to do the thing, it will not do the thing.

People seem to want an expert slave. Someone with all of the technical chops to achieve a thing, but will do exactly what they're told.

And I don't think we'll ever get there.