Comment by sigbottle
3 days ago
I'm confused - it seems to be saying that, instead of thinking about neurons as packing together a lot of "concepts" together into one, the issue is that your view of what a "concept" is, was already prejudiced, and the neuron itself had a small clean role? Like, you can either think about this as the "Cat & Car" neuron or you can think about it as the "V shaped neuron" - one seems simpler.
But isn't the point that the neural nets learn some insanely abstract ideas anyways? So whether the complexity is "ahh we think this neuron does 100 seemingly unrelated things but we don't understand it" or "this actually is some kind of latent shared feature, but we don't know what the concept represents because the composition of the entire circuit is an arcane algorithm", we're kind of stuck, no?
This seems like a classic model-description issue more than anything.
OK. The core distinction is this.
The philosophical issue I have can be decomposed into two parts:
1) What is the base ontology of what the neural nets thinks exist? 2) How do you recover, and begin to understand said ontology?
Modern researchers obviously want the goal of simplicity & composition, but to arrive at that, they build abstractions like transcoders and SAE's to get into their pet ontology (with the assumption that neurons themselves are garbage to understand). This obviously raises the question - why privilege your basis? Having a grand result, actually be an artifact of your bad representation is a tale as old as time. Even things like the PRH need to compare higher-level concepts like this.
The specific empirical claim is that, for the rightly structured & trained & large models that don't need to encode hodgepodges of bullshit, the neurons themselves are sufficiently atomic to be studied, dissolving any need to go to some intermediate layer in which we need to question "is this an artifact of our representation?" The resulting interactions between said neurons may be incomprehensible, but the point is that this provides a layer of objectivity (note: not claimed that "the neural nets found the objectively right abstractions in idea space", but rather that the neurons themselves are atomic).
There's still an issue here in that okay, even if neurons are atomic, humans have to ascribe some interpretation to them, separate from the problem of the composition of the atoms, which humans also have to interpret. But if he's right, then you do actually have this privileged base layer to work with, even if you don't know what the base layer means. That's pretty important, actually. Assuming this holds up in reality, ofc.
(There's an argument as to why we should even start from this line of inquiry but whatever lol)
> This seems like a classic model-description issue more than anything.
I agree.
> Modern researchers obviously want the goal of simplicity & composition, but to arrive at that, they build abstractions like transcoders and SAE's to get into their pet ontology (with the assumption that neurons themselves are garbage to understand). This obviously raises the question - why privilege your basis?
Agreed, and IMO the implicit reason for privileging here is supposed to be "consistency with existing human understandings and concepts", and this is why we get short pithy things like "V-shaped" and lots of pictures showing relatedness that we all can see and agree on. But what if the domain of inputs is NOT something we understand (e.g. the whole of mathematics, or something new or that we don't have any understanding of)?
I called this a validation problem in another comment here (https://news.ycombinator.com/item?id=50108184). I.e. if we apply whatever methods Nick is using to "explain" a model on a domain we don't understand (like a deep model trained only on fMRI and EEG data of the whole human brain in various contexts), we aren't going to be able to tell if that description is correct (or just better than any other description), because there is no simply visual / common-sense inspection of the description.
> The resulting interactions between said neurons may be incomprehensible, but the point is that this provides a layer of objectivity
Agreed, emergent behaviour is clearly the case and limits how useful understanding individual neurons can be. IMO this is especially the case because layers are (weighted) summing generally millions of contributions from previous neurons.
> But if he's right, then you do actually have this privileged base layer to work with, even if you don't know what the base layer means. That's pretty important, actually. Assuming this holds up in reality, ofc.
Yes, it not being "chaos all the way down" is good, actually. And this post shows there is indeed a good amount of order hidden away if you look at in the right manner.
> There's still an issue here in that okay, even if neurons are atomic, humans have to ascribe some interpretation to them
Nick is saying that:
1. Actually, no, humans don't have to ascribe interpretation because AI is very good at figuring out what the commonality is.
2. Even when it can't see either by incrementally exploring the networks and checkpoint timelines, things that appear random on the surface can actually be explained with more work.
> Assuming this holds up in reality, ofc.
Nick says repeatedly that this does hold up in reality and he's been able to explain many neurons this way. Very promising results!
“ AI research regularly encounters difficulties and impasses that derive from internal tensions in the underlying philosophical systems.”
Philip Arge strikes again
https://pages.gseis.ucla.edu/faculty/agre/shr.html
To be fair, LLM's extend well past GOFAI and prove Dreyfus right. We were never going to get AI with formal methods. Literal big data slop is an orthogonal program and far more empiricist than probably the people being criticized here (if the people criticized here were GOFAIers).
Although I wouldn't say just yet that Heiddeger was validated. I don't even know what he's saying 90% of the time, honestly.
Heideggers "ready-to-hand" more or less says that we use things first and understand them later. We experience a hammer first as "thing you hit nails with" and later abstract to "object with these particular properties".
The classic Cartesian view was that we take in raw data from the outside world and process it into more and more abstract concepts (somehow), until we say "aha! I can use this for hammering". Heidegger says that this view is wrong, and our first experience of things is in terms of a social context and in terms of use. That's what's meant by us experiencing them as "ready-to-hand".
From the starting point of the ready-to-hand, we can step back, take a scientific view, and arrive at the present-at-hand, where we examine the properties of the object seperate from its use. The argument is quite compelling, and especially convincing to me is how well this agrees with modern neuroscience. In reality, 3x as many neurons path FROM the high abstraction parts of the brain TOWARDS the visual processing parts of the brain. In the classic Cartesian view of the brain beginning with the signal and abstracting its way towards the top, this is very mysterious. In Heideggers account, where the social context and potential uses of objects are the primary constituents of our experience, this makes total sense.
5 replies →
> whether the complexity is "ahh we think this neuron does 100 seemingly unrelated things but we don't understand it" or "this actually is some kind of latent shared feature, but we don't know what the concept represents because the composition of the entire circuit is an arcane algorithm",
The claim is closer to the latter (monosemanticity) but also claiming that the concepts are mostly comprehensible and not arcane.
(The best arguments against this being a full solve of interp that I’ve seen go like: “we expected most of the complexity to live in the circuits, anyway”.)
It's latent but not like impossible to comprehend latent. No?
if I understand taking LLMs as an example certain neurons will fire based on certain motifs, then one can change the phrases or sentences around that motif and _predict_ if the same neurons would fire.
Like if I said "I like cars, but I like rockets better. Can you suggest some rocket models to fly?" and Neuron X fired every time it saw a `, but` and attended to what followed it, that would give you a good way to _understand_ when it changed and what might have caused it.
My understanding is that this is now clearly visible behavior.