Comment by dleeftink
3 days ago
Why do we want to reduce 'the neuron' and encapsulate it? As a potentially interchangeable feature?
Even when similar weight activations arise across learning regimen it seems we will miss a whole range of interpretations if we just go off 'the text'. The outside context in which a neuron arises is equally, if not more, telling; its genesis cannot (yet?) be quite captured by the weight itself.
The network may reveal its own context and history, but what can ultimately be encoded in its weights that originated outside the network?
Interpretability does try and find neural circuits that consist of many neurons working together, and the paper does discuss that in the later sections. It also discusses observing how a neuron 'grows' during training.