Comment by stillpointlab
3 days ago
Perhaps you can also help me understand why #1 being right and #2 being wrong would be some win for alignment that people on X claim it to be?
My impression, being a total outsider, is that the reaction feels like a bunch of insider bros hyping up their insider bro friend. It feels more like PR than a genuine groundswell behind a truly revolutionary idea.
The big win would be, "Here's the deception neuron, here's the sycophancy neuron, here's the prioritize my own goals over the goals of the user neuron, here's the seek control over my environment neuron."
How much would it cost to find those things, if they exist? For using the blog post's methods to label every neuron in modern arch, Claude and Gemini produce similar back of the envelope math that looks something like, within an order of magnitude of the resources it takes to pretrain the model in the first place. If you have a better way of narrowing down the set of candidates, probably a lot cheaper.
Would that be reliable and useful? Well.
Because if each neuron represents a small set of similar concepts instead of several completely different things, it should be easier than expected to look inside the neural net and see what it's really thinking.
It might still be quite difficult since the network is huge, and just because you know what the individual neurons represent doesn't mean you understand the complicated thoughts they're building. But easier than expected is still good news.
If (2) is correct then you need to train SAEs and other things to figure out what a neural net is thinking. This is very expensive and slow, which is why a lot of neuron-interpretability experiments are done on small networks. The techniques don't scale.
But if (1) is correct, and Nick presents strong arguments that it is, then you can toss out all the indirect stuff like training dedicated interpretation networks. So the techniques can scale and be trustworthy.
Neurons having narrow semantics means you can inspect what a neural network is "thinking" with confidence and even constrain its space of behaviors by activating/suppressing a small set of neurons that have very straightforward effects on its behavior. This is in contrast to polysemanticity where reading the state of any number of neurons may not give insight into what state the neural network is in and controlling the network by activating/suppressing neurons is nearly impossible to do reliably.
> Neurons having narrow semantics means you can inspect what a neural network is "thinking" with confidence
Not really, because of emergent behaviours and also because there are still millions of neurons being summed (with variable weighting depending on inputs) at each layer.
> and even constrain its space of behaviors by activating/suppressing a small set of neurons that have very straightforward effects on its behavior.
You can influence behaviour by doing this kind of thing, yes, but it isn't straightforward at all, and is generally very messy and you get all sorts of unpleasant sides effects (e.g. abliterated models not performing as well).
>Not really, because of emergent behaviours and also because there are still millions of neurons being summed (with variable weighting depending on inputs) at each layer.
Emergent behaviors in terms of seemingly random combinations of features producing unpredictable but coherent/intentional output basically don't exist. In other words, there are no "hidden thoughts" in an LLM encoded in higher order combinations of features. Inasmuch as non-semantic features of the input influence output, it does so in non-semantic ways that are degenerate in terms of the quality of output. The only thing emergent in LLMs is degenerate state leading to a breakdown in tracking relevant features of the input and a reduction in quality of output. But the relevance of this kind of degeneracy to LLM output is at the margins. Tracking activated state predicts subsequent output to a high degree of accuracy.
>but it isn't straightforward at all, and is generally very messy and you get all sorts of unpleasant sides effects (e.g. abliterated models not performing as well).
I expect this has as much to do with our lack of precision in our tools rather than with the concept in principle.
5 replies →