Comment by jacquesm

Sorry for the late response: I've seen pictures like these (but not so nice) in the resnet 50 days, trying to figure out if what was present in the model could be pulled out and explained in terms of features. Now, now doubt I'm not half as smart as the author of TFA but my conclusion was that the big plus of these systems is that (1) it defines features that you would never even think of as features and (2) that these features do not map nicely onto the kinds of features that would be useful to humans. So you end up with an 'explanation' that isn't going to help much if you want to take that explanation in doing something with it. This isn't a 1:1 correspondence with features that you will find useful for steering your classifier and if you start cutting in them you will likely get more than you bargain for. And I think this all maps back to the same underlying issue: you can describe a system to the highest degree that time and effort permit. But to actually understand a system and hence to be able to either optimize a system or engineer such a system from first principles you need more than just a description. You can take all of the molecules that make up a human, describe their position in time and space, you might even be able to put them together in that particular form. But that doesn't mean it is suddenly going to spring to life and that is because you don't actually understand it, you've merely described it. The same goes for a computer that is running some program. You can take it into the lab and dissect it but that won't help you determine what it was doing before you took it to the lab. Dissecting a brain does not tell you much about how it thinks, other than in the most coarse divisions. Another thing that I wonder about with TFA is why he decided to concentrate on classifiers. Classifiers are a different kind of problem than generative AI and I think that that is where the real challenges lie. Classifiers are - at least to me - a lot easier to reason about than LLMs and generative models as a class of its own. I wished I had the vocabulary and the background to make this all more clear but in spite of reading more about AI systems than is probably good for me there are still a lot of mechanisms at play here that I do not fully understand. At the same time: you don't have to play top level violin to hear that someone just hit a wrong note and it is in that spirit that I am writing this comment. I do believe that it would be nice to see classifiers that are fully explainable but I don't quite see the link from the conflation of description and explanation to one where the two are properly separated out. I suspect we will end up with a really good description but just as much explanation as we have today. And that's not necessarily a bad thing: the ability to engineer AI to do your bidding is at least somewhat balanced at the moment by the requirement to train the AI on a very large body of text which always will provide some balance, as opposed to one that is engineered from first principles.