Comment by vidarh
5 hours ago
The most compelling argument to me is "accidentally", due to AI that is made blind to consequences or don't care because it's geared towards a single goal (see e.g. the paperclip maximizer).
We could ask if it is possible to end up with an AI that is smart enough to destroy humanity and at the same time still blind enough to consequences and/or callous enough to do it, but then again we have plenty of examples of humans who have been smart enough to do enormous damage and willing enough to do it.
I don't particularly worry about this, as I believe we'll get plenty of smaller scale warnings if/when we're at a point where those kinds of alignment risks might become a problem, but it is a risk we also shouldn't be blind to.
> We could ask if it is possible to end up with an AI that is smart enough to destroy humanity and at the same time still blind enough to consequences and/or callous enough to do it
Don't make the mistake of anthropomorphizing any A.I. trained under the direction of Larry Ellison.
Anthropomorphising or not is entirely irrelevant to that statement - the question remains the same.