Comment by kypro

AI doomers often talk about these kinds of scenarios often, but we tend to assume if humans were given magical math/physics results which we couldn't understand, or given magical pills by AI that cure all disease, we'd probably just take whatever the AI has given us rather than spend years or decades trying to understand the knowledge/technology before leveraging it.

At some point in complexity – especially if we allow our own knowledge to deteriorate because AI can do the hard work – we will stop understanding the world around us. In the same way one day Native Americans woke up and realised they shared the Earth with people who had magic sticks which they could point at someone and kill them, we will live in a similar world very soon too.

What sticks are dangerous, you will not know. Your existence in the future depend entirely on the AIs not wishing you harm, but you don't know how they work to verify their motivations either.