Comment by antonvs
4 days ago
Yes, (digital) neural networks are algorithms.
You probably know that non-determinism in real-world inference comes from deliberately stochastic features like temperature, from implementation choices like distributed and parallel computation, and numerical effects like floating point precision.
The fact that a neural network's execution can be nondeterministic doesn't mean that the network stops being an algorithm.
The point you're making boils down to speculation that the accidental non-determinism involved in real-world inference is significant enough to undermine or negate interpretability findings based on the algorithms. That would need to be demonstrated.
The CGoL and BB examples demonstrate what might be possible in a worst-case scenario, but they're only analogies in this context. It's far from clear that real models suffer from such effects to a significant degree. You might actually be better off talking about the butterfly effect!
I'm sorry, _I_ need to demonstrate that the accidental non-determinism is significant?
Someone call the FCC, they've been doing it wrong all these years!
I didn't say "you", I said "That would need to be demonstrated." It can't be assumed, for hopefully obvious reasons.
If someone comes up with an interpretability result based on analyzing a neural net algorithm, they'd want to provide some evidence that the result applies to actual executing models.
Happily, we already have good evidence that inference non-determinism doesn't generally invalidate results obtained through analysis of the algorithm. That's not to say there couldn't be pathological cases, but in general, this isn't something keeping researchers up at night.
Yeah, I call bs.
We have had good evidence that the choice of pivot doesn't generally invalidate the nlogn complexity of quicksort.
Except for when it does.
And then we do the work and prove an average of nlogn with a random pivot choice.
Then we know that quicksort is safe to use.
So no, I or anyone else don't need to demonstrate that the non-deterministic outcomes of inference are sometimes invalid. It's the other way around, always has been and always should be. This is "leaded gasoline is safe" mixed with "smoking is not addictive" propaganda.