Comment by D-Machine

For responses, see the linked Twitter thread in other posts here, but also:

https://xxcancel.com/NeelNanda5/status/2110875974296654062#m (and the re-post on LessWrong: https://www.lesswrong.com/posts/x4kvabkF2Jk8GLZp8/linkpost-n...)

TL;DR:

    This is interesting work, but I don't think it's a significant update to the big picture of interpretability. [...] I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc.