Comment by nuancebydefault

If you cross the street 999 times with eyes closed, you might feel comfortable to do it again. But we are ingrained not to do that once. We just understand the risk.

If you do the same with an AI, after 999 times of nothing bad happening, you probably just continue giving it more risky agency.

Because we don't and even can't understand the internal behavior, we should pause, make an effort of understanding its risks etc before even attempting to give it risky agency. That's where all the fuss is about, for good reasons.