Comment by mike_hearn

If (2) is correct then you need to train SAEs and other things to figure out what a neural net is thinking. This is very expensive and slow, which is why a lot of neuron-interpretability experiments are done on small networks. The techniques don't scale.

But if (1) is correct, and Nick presents strong arguments that it is, then you can toss out all the indirect stuff like training dedicated interpretation networks. So the techniques can scale and be trustworthy.