Comment by sigbottle
3 days ago
OK. The core distinction is this.
The philosophical issue I have can be decomposed into two parts:
1) What is the base ontology of what the neural nets thinks exist? 2) How do you recover, and begin to understand said ontology?
Modern researchers obviously want the goal of simplicity & composition, but to arrive at that, they build abstractions like transcoders and SAE's to get into their pet ontology (with the assumption that neurons themselves are garbage to understand). This obviously raises the question - why privilege your basis? Having a grand result, actually be an artifact of your bad representation is a tale as old as time. Even things like the PRH need to compare higher-level concepts like this.
The specific empirical claim is that, for the rightly structured & trained & large models that don't need to encode hodgepodges of bullshit, the neurons themselves are sufficiently atomic to be studied, dissolving any need to go to some intermediate layer in which we need to question "is this an artifact of our representation?" The resulting interactions between said neurons may be incomprehensible, but the point is that this provides a layer of objectivity (note: not claimed that "the neural nets found the objectively right abstractions in idea space", but rather that the neurons themselves are atomic).
There's still an issue here in that okay, even if neurons are atomic, humans have to ascribe some interpretation to them, separate from the problem of the composition of the atoms, which humans also have to interpret. But if he's right, then you do actually have this privileged base layer to work with, even if you don't know what the base layer means. That's pretty important, actually. Assuming this holds up in reality, ofc.
(There's an argument as to why we should even start from this line of inquiry but whatever lol)
> This seems like a classic model-description issue more than anything.
I agree.
> Modern researchers obviously want the goal of simplicity & composition, but to arrive at that, they build abstractions like transcoders and SAE's to get into their pet ontology (with the assumption that neurons themselves are garbage to understand). This obviously raises the question - why privilege your basis?
Agreed, and IMO the implicit reason for privileging here is supposed to be "consistency with existing human understandings and concepts", and this is why we get short pithy things like "V-shaped" and lots of pictures showing relatedness that we all can see and agree on. But what if the domain of inputs is NOT something we understand (e.g. the whole of mathematics, or something new or that we don't have any understanding of)?
I called this a validation problem in another comment here (https://news.ycombinator.com/item?id=50108184). I.e. if we apply whatever methods Nick is using to "explain" a model on a domain we don't understand (like a deep model trained only on fMRI and EEG data of the whole human brain in various contexts), we aren't going to be able to tell if that description is correct (or just better than any other description), because there is no simply visual / common-sense inspection of the description.
> The resulting interactions between said neurons may be incomprehensible, but the point is that this provides a layer of objectivity
Agreed, emergent behaviour is clearly the case and limits how useful understanding individual neurons can be. IMO this is especially the case because layers are (weighted) summing generally millions of contributions from previous neurons.
> But if he's right, then you do actually have this privileged base layer to work with, even if you don't know what the base layer means. That's pretty important, actually. Assuming this holds up in reality, ofc.
Yes, it not being "chaos all the way down" is good, actually. And this post shows there is indeed a good amount of order hidden away if you look at in the right manner.
> There's still an issue here in that okay, even if neurons are atomic, humans have to ascribe some interpretation to them
Nick is saying that:
1. Actually, no, humans don't have to ascribe interpretation because AI is very good at figuring out what the commonality is.
2. Even when it can't see either by incrementally exploring the networks and checkpoint timelines, things that appear random on the surface can actually be explained with more work.
> Assuming this holds up in reality, ofc.
Nick says repeatedly that this does hold up in reality and he's been able to explain many neurons this way. Very promising results!