Comment by asdff
5 hours ago
It just needs to be trained on our laws of thermodynamics and game theory then we are really in trouble. It probably hardly cares about whimsy and love songs in comparison to how it might maximally manage the energy available on this planet and in this solar system.
I'm not convinced you can have an AI intelligent enough to dominate the world if it is trained on a single domain. The intelligence comes from all of the patterns and heuristics it learns across the entire web of all domains. You could train something far more narrow, but it'd be easily overcome by something more general that is tasked with maintaining human objectives.
For a seriously dangerous ASI you'd have to basically train it on everything, and then finetune it for maximal carnage. That's not something the frontier labs are going to do (at least... I hope not), and anyone attempting this with limited hardware will be outpaced by frontier lab AI or the collective of personal agents that aren't misaligned, and they can intercept it and alert on its behavior.
I imagine we'll be getting to a point shortly where anything that is key infrastructure and has the capacity to be accessed on a network will require a permanently running aligned interceptor AI to observe and monitor systems.
We're basically recreating the human immune system in digital form for the entire species.