Comment by JamesSwift

Im responding to the implication that you "have to be ok with good enough" and that somehow this will be a mostly fine autonomous experience with this

> A car that only fails in a road conditions edge case is good enough for the vast majority of cases. You accept that, and issue a manual override for when that edge case pops up

But its just like LLMs. They will never be perfect, and so if you arent actively paying attention and steering the behavior then there is always a risk of spectacular failure. Because if you arent paying attention to "needing to [apply a] manual override" then all of a sudden the AI has `rm -rf /` and you had it in "bypass permissions" mode.