Comment by JoshTriplett
4 days ago
> The bitter lesson is finally coming for the self-driving cars.
Maybe, but the opacity level of models is not acceptable for cars. "Why did it drive under the semi?" "Model said to." "Why did the model say to?" "shrug"
But if the model is an LLM, you actually COULD ask it why it drove under the semi, and it would give you an answer. Now, you may argue that it will just be generating a whole new, backwards-rationalized post-hoc explanation of its own behavior given the logs that it managed to take before the crash. But then I ask you: how do you think a person explains why they did what they did after a crash? I direct you to all of the unsettling split-brain neuroscience literature demonstrating that humans are incorrigible backwards rationalizers who make for unreliable witnesses.
That's a bug, and it would be an awful mistake to replicate that bug rather than fix it.
> Maybe, but the opacity level of models is not acceptable for cars.
That depends on actual performance of the model. I would prefer an opaque model with clearly superhuman driving abilities to a human, or to a non-opaque model with worse performance.
It really doesn't.
https://knowyourmeme.com/memes/a-computer-can-never-be-held-...
No self-driving cars that aren't transparent about exactly how they work. (Ideally, no anything that isn't transparent about exactly how it works.)
This take can perhaps appear to make sense in a situation when clearly superhuman opaque AI models don't yet exist. But once they do, good luck convincing people that they should not save lives or reduce their personal risks, just because they always supposedly need an explanation for any accidental deaths, lol.
In our scenario (self-driving), the one who would be ultimately "held accountable" would not be the computer, or the company, but the person who died after singing a waiver/EULA and getting into a statistically superhuman autonomous car, then having a stroke of incredibly bad luck. Such events will happen, but they will be very rare.