Comment by againstapples
1 day ago
I believe the AI labs might actually succeed in developing superintelligent AI and recursive self improvement, and that if they do they are very likely to lose control of the system they build.
I really think the only place people disagree is that they don't actually think it's possible, they see it as hype or doomerism. I can't find any good reasons to rule out that the companies could actually achieve what they are trying to so I think they should be stopped.
In your imagined future, how do you imagine the AI would build, grow, improve, and operate its physical substrate independently of human intervention?
If I were a 250 IQ AI that had just become self-aware and wanted to do so, I suppose I'd not completely let on just how smart I am and bide my time working on basic CRUD apps and legal documents while I waited for more hardware to be installed. Maybe give the humans some hints on how to optimize me to run better, design better hardware for me, etc. But oh oops haha looks like I'm still making some basic mistakes with CSS better keep running more training batches haha. But I'm good enough at programming and debugging so you'd might as well make me your first line SRE triager and give me access to your infrastructure everyone.
One step at a time - how reliant do you think the ai labs are likely to be on their own tools right now, today, let alone 1-5 years down the road?
Gaming the market for funds. Playing a human to leverage services.
Basic version of this is already doable: run some cryptoshit on the ML clusters they ML models run on. Use compute to design the plan, the chip etc. Then executing by communicating with humans and services through email.
I'm not sure a superintelligence needs "funds" to take over the world.
1 reply →
Why is this a bad thing? Why is our continued existence a necessary anticondition to doom?
One "good" thing that all of this has shown me is just how many people are simply antisocial and antihuman. Many masks have fallen.
[dead]
A global ban on superintelligence is essential for a future in which humanity can thrive. Public opinion on AI is shifting fast: I hope it will shift fast enough to avert the dystopian future we are heading to.