Comment by zug_zug
1 day ago
I think it's pretty apparent that current-version LLMs won't wipe out humanity. But when you reflect that these GPT models are just token-predictors were never engineered optimally, it seems entirely plausible to me that there are multiple order-of-magnitude optimizations yet to be made.
If such were achieved, the model would almost certainly be smart enough to make itself smarter, and hack as much compute as it could possibly want.
So if we ask what would be done by an intelligence (human or otherwise) that is beyond human comprehension, it would be pure hubris to say we know for sure. We can scarcely control the models we have right now (e.g. hugging face attack). But given our whole society is mediated by technology, an superhuman intelligence could certainly collapse the government.
Certainly collapse the government? How?
Possibly in much the same way humans currently do this: bribing and lobbying, misinformation campaigns, cyber attacks on elections, blackmail. They're already being used for some of these, just perhaps not autonomously.
How does this rogue AI pay for it's electricity?
Most straightforwardly: literally taking control over the electricity generation facilities. Less straightforwardly: bitcoin (and other cryptocurrency) miners. Even less straightforwardly: the same way the OpenAI et al. pay for their electricity, by selling the capabilities of its AI for interested parties to use.
With crypto.. that it makes on Polymarket-like ways, or creates its own content? Or blackmail humans? :-) I'm not entirely serious with my comment and do agree with you that there are lots of more immediate safety topics we should address before worrying about the AI becoming self-aware. That said, finding ways of accumulating valuable resources could be intermediate activities the AIs will attempt to do to complete it's 'goals' even when they're human-set!