Comment by spolitry

what if constant improvement in AI results in them breaking modern public key cryptography, mathematically? What if it starts proposing novel forms of cryptography and breaking them at faster and faster rates? Nobody will have a formal ceremony that "lets the AI control the development of cryptography". It could provide humans with a form of cryptography it knows we can't break but it knows it can break. Humans might even request such cryptography, in the hopes of convincing others to use their backdoored cryptography. The machines will be aware we can not verify if some urgently needed replacement cryptography is backdoored by it. Any internet connected instance that is smart enough can log in anywhere.