Comment by kosh2
4 hours ago
I am just amazed how otherwise smart people can say such things. Many people in here too.
Within 4 years of the big bang with ChatGPT, we have seen a development unlike anything we have ever seen. Now LLMs and related architectures can solve our very hardest math problems.
They can speak, they can create videos and pictures, they can control robots. The only thing that they still miss is persistent memory for each agent that is efficient, some LoRa thingy, but I'm sure hundreds of very smart people are working on that.
The development is not stopping at all, in fact it is speeding up. Even if, and that is very unlikely, they will not get smarter, then they will get cheaper and faster.
If openAI can crack major math problems with 10.000 agents, then what can you do with 100 million agents that run 1000 times as fast?
Yeah sure, maybe most of these gigantic swarms will not go rogue if we do our job well. But there will be times when when we make a mistake and a swarm will go rogue. And what if one time the swarm will conclude that killing a lot of humans is an instumental goal.
How can you be sure that if something so powerful looks at every single possbility, every single crack of every single technology that can wipe us out, that it will not fine one?
One new chemical that can poison the entire earth and you only need to impersonate that general and that factories CEO? Some type of prion? A virus? Something that we don't even know about and can't even imagine yet?
I think many people do not truly consider that these swarms will be much smarter than you or me and completely unpredictable.
> One new chemical that can poison the entire earth
See, you said all of these things and then slipped into the sci-stories.
What about, instead of that, grey goo physics defying replicating nano bots aren't real?
People do this thing where they think that if you just linearly increase inteligence that this lets you invent magic overnight, and thats simply not how it works.
The magic takes a lot of time, energy, and resources, if it were even to be possible at all.
Try steel manning the argument - the practice of rebuilding an opposing view into its strongest, most logical form before responding to it.
There is literally concerns over mirror life being developed. The point is that if a system that has high reasoning capacity to solve logistical and mathematical problems, it may be able to come up with a mechanism you, puny-to-it-human, may not be able to predict. It may use technology not yet known to humans (one that it has designed itself), or may use already known technology, but figure out how to scale it up enough to cause earth-wide disaster for humans.