Comment by phyzome
4 days ago
16,000 words into the essay: "What I mean by clean"
...and it still doesn't answer it. What is this actually about?
4 days ago
16,000 words into the essay: "What I mean by clean"
...and it still doesn't answer it. What is this actually about?
It means that NNs don't (as previously thought) smoosh unrelated concepts together into the same neurons, which would have made interpretability very difficult.
For instance, if you had a single neuron that fires for "puppies" and "murder", it would be much more difficult to tell if the model is thinking something nice or something not so nice.
"clean" means not hopelessly incomprehensible which is/was the prevailing consensus among interpretability researchers
the idea that most inexplicable circuits start making sense through the lens of recursive generalization seems intuitive to me for image diffusion models but I'm not so sure about language models because of the "once removed" nature of them - I haven't made it to the bottom of the article yet so maybe this gets addressed
It means that modern neural networks are way more interpretable than older ones. We can understand how they work way more easily than earlier ones.
For context: Goodfire has expertise in interpretability through a Sparse Auto Encoder technique of their own
https://www.goodfire.com/blog/a-practical-guide-to-saes
Sparse autoencoders are not reliable. Ask your favorite AI. For example, https://proceedings.mlr.press/v267/kantamneni25a.html
A lot of the mech interp techniques are glorified notions of probing. If models were generally interpretable with respect to the things we cared about, we'd know know when they're hallucinating, for example.
Identifying that a common neuron or set of neurons fires for golden gate bridge can be done only because you already know that you're looking for golden gate bridge. If models had recovered some new truth (e.g., the silver door road), you wouldn't be able to elicit it without knowing to look for it a priori.
2 replies →
I think "clean" means something like each neuron encodes exactly one concept and it's possible to discover what the concept is.
But they don't. That's the whole reason they work in the first place.
Every neuron in the layer encodes fractionally something about the concepts embedded in the whole thing. It's more like a hologram than a series of connected facts.
Nick's claim is that that's the fundamental misinterpretation. Each neuron eventually ends up mostly doing one main thing (and the rare case that some neurons do several related things). Neurons compose to embed ideas which is a starkly different interpretation than them being holograms and fractional encodings.
4 replies →
Doesn't "orthogonal" convey that meaning more crisply and precisely?
Orthogonal means something different in mechinterp, so it'd be an overloaded term. Usually orthogonal would imply that there are actual vectors under consideration, which isn't the underlying claim.
Using "orthogonal" here would have the opposite effect, as it would impart a far more specific meaning which is also incorrect in this case.
by clean it means the opposite of "messy"
https://already-clean.goodfire.com/assets/superposition-circ...
That's the picture under "some are both dogs and cars" in the artcle
It's actually a clean shape, if you disregard the color and texture.
Clean means no alchemy.