Comment by ACCount37
1 day ago
Humans are dominating the environment by hopelessly outsmarting everything in it. Applied intelligence is extremely powerful.
Humans, however, are not immune to being hopelessly outsmarted themselves.
And what are we doing with AI now? We're trying to build systems that can do what human intelligence does - but cheaper, faster and more scalable. Multiple frontier labs have "AGI" - a complete system that matches or exceeds human performance in any given domain - as an explicitly stated goal. And the capabilities of the frontier systems keep advancing.
If AGI actually lands, it's already going to be a disruption of everything. Already a "humankind may render itself irrelevant" kind of situation. But at the very limit - if ASI follows?
Don't think "a very smart human". Think "Manhattan Project and CIA and Berkshire Hathaway, combined, raised to a level of competence you didn't think possible, and working 50 times faster than human institutions could". If an ASI wants power, it will get power. Whatever an ASI wants to happen will happen.
And if humanity isn't a part of what it wants? 10 digit death toll.
Even if LLMs don't become AGI (and I don't think they will), LLMs are potentially superb disinformation generators able to operate at massive scale. Modern society was already having difficulty holding onto consensus reality. "AI" may be able to break it.
Don't think "smart human". Think about a few trillion scam artists who cannot be distinguished from a real person except by face to face conversation.
Your every avenue of modern communication and information being innundate by note-perfect 419 scams, forever.
This hierarchy-of-intelligence concept is meaningless.
You might as well have said "intelligence is meaningless". Obviously false.
No, I might not as well have said that. Let me ask you this: huh?