Comment by ben_w

Every random group of intelligent humans who is motivated to seek world domination is constantly in a fight with all the other groups of intelligent humans who are also motivated to seek world domination. They fight as peers.

Despite this, a minor technological and social advantage allowed one small sheep-filled rock in the north-east corner of the Atlantic to hold 25% of world GDP, population, and land area (though not all at the same time), and to hold on to >20% of world GDP for about 70 years.

For better and for worse AI, as it currently exists, is not shaped like our minds. Better for us, because machine learning needs a lot of examples, so if we invent a different behaviour or create a different environment, a machine would struggle to cope with it, this is also why they needed to read basically the entire public internet to get good; worse for us, because the hardware it runs on comes with an absurd speed advantage, which is why they are able to read the entire public internet and by extension when they got good they got good at basically every subject at the same time.

And:

> Edit: And when somebody asks to be directed, that's called seeking advice. What I meant is having the legal authority to make people do things while not acting on other people's behalf.

Great, but so what? We have lawyers doing this, too. Trump's tariffs on the island occupied solely by penguins looked suspiciously like it came out of an LLM (even despite them being worse then than now).

And LLMs are demonstrably more persuasive than human writers, so it's not even particularly unbelievable at this point (even if it ought to be, c.f. chatbot psychosis) that an LLM will simply convince people to give it the right to form a government and wield such powers.

Agree. We already have a planet with eight billion human level intelligent beings on it. Despite some close calls in the past, none of them have yet managed to wipe out humanity.

> Even this would require monumental advances in physical compute and HMI, but I wouldn’t rule this out within 20 years.

Would it require so much? Cochlear implants exist if you care about HMI, and compute is already doing "reasoning"[0] faster than humans do snap judgements. Actually winning in an adversarial environment against the best human political minds (let alone economic or military) will be harder, but I would be (pleasantly!) surprised if it takes 10 years, let alone 20.

> I could see Elon Musk having this kind of idea. But even in this case, I don’t think “nuke everybody” would be the highest-ROI strategy advised by the AI.

I hope a sensible AI would not. However, Musk's LLM called itself "MechaHitler", which suggests his training process does not avoid such things. If his in particular becomes dominantly competent, he has also spoken of wanting more Optimus robots than there are humans[1], has called it a "robot army" (and his justification for even more Tesla stock was to keep influence over said army[2]), and he has spoken of factories on the moon churning out space-based data centres by the petawatt which is enough that if placed in LEO it could have about the same effect on Earth's climate as all anthropomorphic greenhouse gasses currently in our atmosphere (and that's even if not weaponised, that's just simply by having that much stuff that close).

On the plus side, Musk's got a huge gap between what he says his AI will do and what they end up doing on those timescales, so hopefully we'll be safe from him: https://en.wikipedia.org/wiki/List_of_predictions_for_autono...

> This is probably closer to the imagined doomsday scenario, but again I find it hard to imagine that such an AI would advise nuking everybody.

I don't. First, because AI being "better" doesn't make it infallible; and second, an argument shaped like this has historic echoes with actual nukes and the cold war: if the AI is competent to take over conventionally, and everyone knows this, then every other nation with nukes will use them, therefore the AI should use a first strike. (Note: the argument does not need to be perfect or flawless in reality, it only needs to be the kind of thing which humans have found convincing, and far too many humans have argued their side would win a nuclear exchange).

> If AI takes over the world, I imagine it will be because of the production efficiencies it enables. Which is a different kind of takeover - one that would eventually have broad benefits for many people - but would likely go through a period of major instability due to labor displacement.

This is my best case scenario. Some people also object to this, they like their careers and fear what retirement will do to their sense of self-worth, but I suspect this can become an opportunity for flourishing in other ways if, and only if, it is handled correctly.

[0] something something swimming submarines, I don't care about using anthropomorphic language myself, but there's always someone ready to complain if anthropomorphic language doesn't get scare-quotes

[1] "And, and then there'll be obviously robots in industry making stuff. And so I mean, I think the ratio of humanoid robots to humans will probably be at least two to one, something like that. One to one for sure. So which means like somewhere on the order of 10 billion humanoid robots, maybe, maybe 20 or 30. And so then it's like, okay, well let's say, you know, you kind of make, let's say the build rate is, I think the build rate will be probably something ultimately like a billion a year." - https://www.elonmuskarchive.org/video/tesla-shareholder-meet...

[2] “If we build this robot army, do I have at least a strong influence over this robot army?” he told investors. “Not control, but a strong influence… I don’t feel comfortable building that robot army unless I have a strong influence.” - https://futurism.com/robots-and-machines/elon-musk-remarks-r...