Comment by Fripplebubby
3 days ago
I think the paper says that while there do exist dog neurons that carve out the difference between a "Boston terrier" and a "terrier", those neurons actually only get to carve that fine line because they're part of a whole flow control system where activation has already flown through much coarser grained discriminator neurons (like a "V" neuron), including sometimes through the negative side of activations, before the terrier discriminator ever played a role.
So, if you try to draw a circle around the "dog" part of the network, do you circle the Boston terrier vs terrier neuron only? You'll be missing most of the information that represents the "dog" category and that same dog category is also involved in shunting activation away from things that are toward-being-a-dog in whatever sense is being discriminated in that moment.
You’re only considering the front half of the network though, classification is a helpful example for explaining the concept - but if it follows through to the output then you could quite easily imagine just shunting some activation from one branch of the representation tree to another to steer behavior. Which we already do with steering vectors - but if you could read off the actual topology you could do that with much higher fidelity.