Comment by D-Machine

> This seems like a classic model-description issue more than anything.

I agree.

> Modern researchers obviously want the goal of simplicity & composition, but to arrive at that, they build abstractions like transcoders and SAE's to get into their pet ontology (with the assumption that neurons themselves are garbage to understand). This obviously raises the question - why privilege your basis?

Agreed, and IMO the implicit reason for privileging here is supposed to be "consistency with existing human understandings and concepts", and this is why we get short pithy things like "V-shaped" and lots of pictures showing relatedness that we all can see and agree on. But what if the domain of inputs is NOT something we understand (e.g. the whole of mathematics, or something new or that we don't have any understanding of)?

I called this a validation problem in another comment here (https://news.ycombinator.com/item?id=50108184). I.e. if we apply whatever methods Nick is using to "explain" a model on a domain we don't understand (like a deep model trained only on fMRI and EEG data of the whole human brain in various contexts), we aren't going to be able to tell if that description is correct (or just better than any other description), because there is no simply visual / common-sense inspection of the description.

> The resulting interactions between said neurons may be incomprehensible, but the point is that this provides a layer of objectivity

Agreed, emergent behaviour is clearly the case and limits how useful understanding individual neurons can be. IMO this is especially the case because layers are (weighted) summing generally millions of contributions from previous neurons.

> But if he's right, then you do actually have this privileged base layer to work with, even if you don't know what the base layer means. That's pretty important, actually. Assuming this holds up in reality, ofc.

Yes, it not being "chaos all the way down" is good, actually. And this post shows there is indeed a good amount of order hidden away if you look at in the right manner.