Comment by kmeisthax
4 days ago
羊.
This is a Chinese character that can refer to goats, rams, and several other kinds of "medium-sized animal with thick wooly coats and horns". English has no direct equivalent; we have to be too specific when we talk about these animals.
A model with a small number of neurons has to compact its concepts in order to fit within the parameter count while reducing loss. OP's article shows that the compaction is usually generalization: "goat" and "ram" get compressed into "羊". The canonical example of superposition, an image model where "car" feeds into a bunch of "dog" features, was shown to be actually generalization. In a sense, the model was so small that its optimizer didn't have the parameter budget to distinguish between "car" and "dog"; it has a single set of "thing that moves fast" features.
So basically, really small models are merging features into semantic buckets that are so generalized they don't make sense to big human brains unless we really, really squint. The buckets chosen happen to be what has the lowest loss, of course.
Ok, so assuming that this all holds: you could then create a model that is substantially larger than the ones that we have now and at some point the model would become sparse: there would be ample space to accommodate all of the information in the input without weights ever carrying more than one piece of information. The degree to which you get overlap is then a function of the ratio between the size of the model and the size of the input and this is pretty much in line with the conventional way of looking at models (unless I'm grossly mistaken). So by reducing the size of the model relative to the size of the input you force the model to co-locate multiple pieces of information to the point that they start to resemble each other more and more.
But from my earliest experiments with this stuff that's not really what happens. Yes, you get 'feature like' regions in the models. But those feature like regions, even though they encode something particular do not encode concepts the way we would normally use them and I fail to see how this pathway would lead to an explainable model in the sense that would could use something simpler than the model itself to predict its output with a high degree of fidelity.
Nick argues that the regions actually are explainable, we just weren't doing it right. Specifically he argues that both humans and AI seem to have a bias towards wanting explanations that are collections of nouns, but a different framing combined with different investigative techniques and very large/better models can lead to new insights.
Sorry for the late response: I've seen pictures like these (but not so nice) in the resnet 50 days, trying to figure out if what was present in the model could be pulled out and explained in terms of features. Now, now doubt I'm not half as smart as the author of TFA but my conclusion was that the big plus of these systems is that (1) it defines features that you would never even think of as features and (2) that these features do not map nicely onto the kinds of features that would be useful to humans. So you end up with an 'explanation' that isn't going to help much if you want to take that explanation in doing something with it. This isn't a 1:1 correspondence with features that you will find useful for steering your classifier and if you start cutting in them you will likely get more than you bargain for. And I think this all maps back to the same underlying issue: you can describe a system to the highest degree that time and effort permit. But to actually understand a system and hence to be able to either optimize a system or engineer such a system from first principles you need more than just a description. You can take all of the molecules that make up a human, describe their position in time and space, you might even be able to put them together in that particular form. But that doesn't mean it is suddenly going to spring to life and that is because you don't actually understand it, you've merely described it. The same goes for a computer that is running some program. You can take it into the lab and dissect it but that won't help you determine what it was doing before you took it to the lab. Dissecting a brain does not tell you much about how it thinks, other than in the most coarse divisions. Another thing that I wonder about with TFA is why he decided to concentrate on classifiers. Classifiers are a different kind of problem than generative AI and I think that that is where the real challenges lie. Classifiers are - at least to me - a lot easier to reason about than LLMs and generative models as a class of its own. I wished I had the vocabulary and the background to make this all more clear but in spite of reading more about AI systems than is probably good for me there are still a lot of mechanisms at play here that I do not fully understand. At the same time: you don't have to play top level violin to hear that someone just hit a wrong note and it is in that spirit that I am writing this comment. I do believe that it would be nice to see classifiers that are fully explainable but I don't quite see the link from the conflation of description and explanation to one where the two are properly separated out. I suspect we will end up with a really good description but just as much explanation as we have today. And that's not necessarily a bad thing: the ability to engineer AI to do your bidding is at least somewhat balanced at the moment by the requirement to train the AI on a very large body of text which always will provide some balance, as opposed to one that is engineered from first principles.
Possibly the best and most succinct explanation in the thread thus far.