Comment by d_silin
13 hours ago
To add a bit more constructive feedback, think of the "AI cornucopia" and "Superintelligence destroys humanity" as opposite ends of all possible outcomes distribution (low probability event).
The most likely one is the "business as usual, but with AI" - some things will get better, some things will get worse, but overall state of affairs will remain mostly the same.
The people who've thought the most about this put it differently:
Think of a new, superintelligent model as if it was a new v1 Starship launching for the first time, with a full fuel tank. On the one hand, rockets have existed for some time, and some have gone to space successfully, including by this company.
On the other hand, this is a tube of metal full of highly explosive liquid going faster than most human objects ever go, for the first time ever in this novel and state of the art configuration.
If someone said, "really, the first Starship exploding is just at one end of the probability distribution, where the other is that everything goes fine and all its passengers have a nice trip in space," would you get on that rocket?
Or, more aptly, if you and every other living human was already on that rocket, would you push the launch button?
The analogy works because superintelligence is, like rocket fuel, an extremely powerful force that has a default tendency to break containment and go boom (consume lots of energy and heat and matter in a chain reaction, to pursue more intelligence to pursue whatever goal it is pursuing.)
> superintelligence is, like rocket fuel, an extremely powerful force that has a default tendency to break containment and go boom
what evidence do we have this is the case?
We can learn from history: Albert Einstein famously tricked humanity into building nuclear weapons for him and was only prevented from wiping out all sentient life by the Princeton IAS Board of Alignment who published a very compelling blog post about realigning the A-bomb contra paperclips.
1 reply →
This is the sort of pseudo-scientific reasoning by analogy that leads Elizier Yudkowsky to argue we should be terrified that an unfriendly ASI could rapidly develop "diamondide" nanobot viruses, because it will be super-intelligent and super-intelligence can do anything by first-principles:
Thought experiments ungrounded by any realistic technological constraints or scientific evidence are pretty much useless for actual forecasting except as an exercise in sci-fi worldbuilding.
https://www.lesswrong.com/posts/bc8Ssx5ys6zqu3eq9/diamondoid...
https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...
Eh. An intelligence with perfect knowledge of the entire universe, unlimited memory, and infinite processing speed, would have a tendency to go boom. But the real world has limits, and intelligence, even superintelligence, does not equate to godhood. Some tasks are still hard no matter how smart you are.
Plus, we have no real reason to think LLMs are anywhere close to AGI or ASI. So arguments like these are just distracting from the very real, very present danger that LLMs pose: information breakdown, societal collapse and environmental destruction. In other words, this is criti-hype.
The kind of person who thinks AI contributes to information breakdown rather than information diffusal has no theory of how information spreads. The other claims like societal collapse and environmental destruction are still FUD but slightly more possible
I would like to classify this as “tail risk fallacy” because it exploits humans natural tendency to be risk averse. No matter how small the tail risk is, the fact that it exists can be justified to stop whatever you want to stop. A thoughtful person might say there’s a threshold at which you might want to care about it but that’s not how ideas spread.
Take climate change: you can spread whatever dubious claims about tail risk being whatever probability with amateurish models and use that to justify degrowth. This scheme actually works - most people in the west are already brainrotted by the tail risk discourse in climate change.
We don't need tail risk to want to mitigate climate change since the mundane damage it is causing is already here right on schedule.
Not to mention gasoline has become unaffordable so switching to the new technology has become economical.
Current AIs are closer to bottle rockets than to the Starship on the intelligence scale. Some property damage already happened, but you can't master the art of rocketry without trial and error.
But will “business as usual, but with AI” justify the current capital expenditures? I think the market is pricing things as being closer to “AI cornucopia”.
What happens if/when we don’t deliver?
> What happens if/when we don’t deliver?
I wish we had real journalism, the AI labs CEO should be asked that question in every single interview
Market bubble will pop, not the first time and not the last, most likely after OpenAI and Anthropic IPOs.
My guess is after Anthropic and before OpenAI