Comment by lukewarm707
3 days ago
Let me quote the opening paragraph of the paper:
"After many months studying neurons and circuits in image and language models, I’ve come to believe a network’s own mechanisms are its legible building blocks, and that the outlook for mechanistic interpretability may be far better than I’d feared. The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread. What looked like arbitrary packing was mostly a motif I’ll call recursive generalization, which is much less concerning."
> > "a network’s own mechanisms are its legible building blocks"
Indecipherable.
> > "The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread."
State the claim in plain english. Use the active voice.
"Concepts do not overlap arbitrarily."
> > "What looked like arbitrary packing was mostly a motif I’ll call recursive generalization, which is much less concerning."
Jargon!
"Concepts overlap in a recurring pattern."
It's a research paper targeted at other researchers; you do need to understand some of the field for it to make sense. It's not written to be MechInterp 101.
But it's not bad writing. It's actually very clear. I'm not an professional AI researcher, I just read papers for fun, but I found everything in Nick's article clearly articulated and easy to understand. It's actually one of the more digestable papers I've come across this year.
Here's a translation to easier language:
> "a network’s own mechanisms are its legible building blocks"
To understand this you have to know that up until now attempts to figure out what's going on inside neural networks have relied on indirect techniques, in particular by training another neural network to "unpack" what's happening in the first. This was driven by the belief that a deep neural network isn't directly readable (legible) because the neurons do so many unrelated things simultaneously.
Nick is saying that actually, you don't need a second neural net. You can just read out what's happening directly from the first's own "mechanisms" (arrangements of neurons and weights between them).
> "The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread."
In simpler English: we thought individual neurons did lots of unrelated things, but that's because the standard evidence we had for that belief was misunderstood.
"Concepts do not overlap arbitrarily" and "Concepts overlap in a recurring pattern" don't mean the same thing as what he's saying.
"a network’s own mechanisms are its [...] building blocks" is tautological.
> > "the standard evidence we had for that belief was misunderstood."
The author did NOT misunderstand the evidence of polysemanticity and superposition. Rather, the explanatory hypothesis was incorrect.
The word "legible" is the critical part of that sentence.
1 reply →
There is indeed a huge amount of useless jargon and IMO deliberate obscurantism in deep learning research.
Sometimes it is to defensively ward off critics, sometimes it is to woo the naive and ignorant, and sometimes it is because the basic ideas are really just incredibly simple and based on simple mathematical intuitions or hunches.
In the latter case, papers and authors appeal to some obscure or overly-complicated technical theory (which usually doesn't actually apply at all to current deep models as they are actually implemented, e.g. infinite-width networks and other such purely theoretical abstractions) to cover up the fact that they really just had a lucky intuitive guess (and likely that twenty other intuitive guesses just didn't pan out). This appeal to theory is often self-deception as much as it is appealing to academic reviewers.
I think this article is a bit of Nick wooing himself and others here... I'll wait for his supposedly forthcoming paper to see if there is any real meat in it.
I would even say "Arbitrary packing" was a weak conclusion to begin with.
'Arbitrary' was the wrong word, projections encode information.
The more charitable 'random' is a null hypothesis. It is formally impossible to prove. It is uncomputable.
So it is not surprising to see someone reject the null.
The fact that it was given any credence suggests to me deep problems with interpretability or ai safety in general.
Take the headline here: "interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building."
I deny even the existence of alignment. In my view, the field of 'alignment' works on a problem that does not exist.
Alignment is a reference to something that does not exist in the model: social norms. It will never be intrinsic to the model. You will not find it there.
'Neurons' are not primitives either, they are not intrinsic to the model. 'Neuron' is a human-constructed category. It is easy to just make up intrinsic properties of neurons and get away with it.
Yes, "alignment" is very wishy-washy and often does mean pointless moralizing / following of social norms. Obviously thinking about alignment in that sense will be a waste of time, since we don't have particularly consistent or universally applicable ethics (or frankly even general knowledge) in the first place.
But if the term is understood in a more narrow and applied sense, it is a useful shorthand (or can be). Jon Stokes said it nicely on Twitter recently:
https://xxcancel.com/jon_stokes/status/2110404230053802443#m
1 reply →
[flagged]