Comment by canjobear
2 years ago
Yes, a pretty fundamental technique and one of the earliest. It lets you determine which layers contain what information among other things.
2 years ago
Yes, a pretty fundamental technique and one of the earliest. It lets you determine which layers contain what information among other things.
The downside is that it's a supervised technique, so you need to already know what you're looking for. It would be nice to have an unsupervised tool that could list out all the things the network has learned.
Anthropic has published some cool stuff in that direction: https://transformer-circuits.pub/2023/monosemantic-features
Whoa, this is super cool! I can imagine if we had something like this for ChatGPT, we could use it to do some serious prompt engineering. Imagine seeing what specific neurons you were activating with your prompt, and being able to identify which word in your prompt was triggering an undesired behavior. Super cool stuff, excited to see if it scales