Comment by yurimo

I'll say while this is not very realistic now, I believe there isn't a 0% chance of a rogue AI traversing the net and placing itself somewhere(as much as this sounds like a scene from beloved cyberpunk franchises). As models become more efficient and capable, and hardware becoming better, hosting a model should be feasible on smaller devices (by compute and memory). It is possible it can be done in a distributed manner, where distribution is possible on different devices running the same OS or through some interface to collate different OS types.

I doubt the agent will be coming after your smart kitchen toaster, but a cluster with weak security is possible and tasty. This year there was a pretty loud demonstration of what a runaway reward hacking will do. I encourage people to read up on "stop button problem" i.e.: https://www.youtube.com/watch?v=3TYT1QfdfsM

While I think the article exaggerates to a degree, is it really so improbable that an agent seeking to avoid a shutdown and aiming to complete some goal and to get a reward will not attempt to escape if it concludes that it has failed, and humans will kill the process and hence prevent it from completing the task?

Note: I think siloed sharded inference is possible. But unnoticeable large-scale training probably not, at least with the current architectures.