Comment by bloaf
7 hours ago
I think the central premise looks more like this:
Biology has been trying to grey-goo the world for billions of years, but it turns out the world is not something so trivial.
He is pushing back against the folks with pure CS backgrounds who think that computers are all there is. Its a form of magical thinking unique to programmers who live in a world where speaking the right words to a machine is enough to impart your will on the world. Believe that strongly enough, and you fall into the trap of thinking that a sufficiently smart entity could speak the words "let there be light" and it would be so.
The author is pointing out that speaking the words is insufficient. To end humanity there must be an execution phase. The author is correct to point out that acquiring superhuman intelligence is not some guarantee that you will have or obtain the resources necessary to make that happen, in the same way that genius generals still lose to ordinary ones, and the best-laid plans are oft to go awry.
There certainly are risks, but 10% risk of extinction in 10 years is not one of them.
Biology is extremely slow, try to extrapolate the previous 3 years of AI progress 10 years into the future, it's not a comparable trajectory.
Spoken like someone with a somewhat limited understanding of what it actually takes to calculate the fitness function of arbitrary organisms in an arbitrary environments.
Yeah it takes long because natural selection is a blind process starting from absolute zero and is only concerned with fitness, AI is a human guided process, has all of human progress as a starting point and is concerned with any number of goals we can create reward signals for, there's almost nothing you can extrapolate from one to the other.