Comment by kaibee
6 hours ago
> is smart enough to know they need to stop because they'll kill everybody by continuing.
Yeah like when Tobacco companies learned that smoking... well, hmm, well the fossil fuel companies when they learned about climate change they...
Well, I'm sure this time executives will prioritize the common good.
If we're following that logic I really don't want to see what the misaligned internal research models look like
The, ahem, good thing here is that the ASI disaster scenario "everyone dies" includes AI executives.
I think it doesn't matter. Most cancers don't stop growing when they're about to kill their hosts.
AI companies know they have to constantly push further, or they'll get outcompeted and lose their wealth, and nobody agrees on where the line is for "so dangerous it threatens humanity" (and when they try to be conservative about it, everybody screams "marketing stunt" and rushes to competitors).
If a single company decides "enough is enough" and stops chasing the state of the art, everybody goes to their competitors, they lose the money faucet, their employees go work for those competitors. The competitors also (usually) know they're building an existential risk machine, but they think they can push a little further, and they don't want to go out of business either.
This equilibrium can last for quite a while even if everybody involved thinks it's a threat to their lives.