Comment by jordanb
4 hours ago
OK understood. But you understand this makes them sound like the type of person who doesn't believe any of the "worldly concerns" are worth addressing because "the end is near" right?
4 hours ago
OK understood. But you understand this makes them sound like the type of person who doesn't believe any of the "worldly concerns" are worth addressing because "the end is near" right?
There's no logical error here if the end is actually near.
So it all hinges on an empirical disagreement you have with them. There's nothing particularly unserious about that.
“It all hinges on [an entirely presupposed conclusion that we can’t prove is any more likely than the opposite]” is about as unserious as it gets lol
Maybe compare the argument that AI has large detrimental environmental impacts to the argument that it has economic impacts. Why would the environmental impacts even be a major argument vs. the worldly economic concerns? Because there is climate science predicting extremely negative effects on humans from warming, e.g. "the end is near" on limiting climate damage. The environmental argument wouldn't have been reasonable to bring up in the 1950s if AI had gone according to the earliest optimistic plans and not required giant data centers.
There is quite a lot of mathematical research into agentic behavior that suggests a combination of instrumental convergence and the orthogonality thesis make it very likely a superintelligent agent will have arbitrary goals that lead it to attempting a takeover of Earth's resources to achieve them.
There can't be a science of superintelligence because it doesn't exist yet, but the best theories I have read seem sound, similar to how 19th century theories of anthropogenic climate change turned out to be sound.
Please link some of the mathematical research which concludes takeover of Earth’s resources by a superintelligent agent is “very likely.”