Comment by questionable_ju

Excellent article, I really enjoyed it.

I especially enjoyed this subtle joke: "We think the weak activations usually don’t matter, but the strong positives and negatives matter a lot. For instance, Santa lives on the negative pole."

I can only imagine how much joy he felt writing that.

As for serious comments on its content, I'm confused by the people who considered it difficult to read. I think the ideas presented were very elegant and fairly simple to understand. Considering neurons as gates and weights as votes was a nice way to look at it, for example. I didn't feel like the post was unnecessarily long either.

One thing that stands out to me is how ridiculous it is in the first place to try to assign a short, natural language description to the function of any particular neuron. Wittgenstein already talked about how humans seem to use words in very specific ways while being terrible at exhaustively describing why.

An example is the word "game". It's very easy for even a child to distinguish between "game" and "not a game". But it's incredibly difficult to write down a precise definition of "game" that includes everything normally considered a game and excludes everything that isn't considered a game. Writing down the function of one specific neuron seems to run into this issue. When you write down what the exact function of any neuron is, there's a good chance your clunky natural language description misses some fine nuances.

A reasonable analogy, in my view, would be that neurons could represent something like the atomic components of intuitions or "vibes". Similar to the example of "game", the feeling of knowing what is or isn't a game is composed of what Wittgenstein called "family resemblances". Your brain doesn't contain a precise description of "game", but upon encountering something, you can quickly compare it to other examples of games you've seen and intuit whether something has a "game-like vibe".

Another notorious example would be "what is a woman". For some reason, everybody has very strong feelings about what is and isn't a woman, but people have a terribly difficult time perfectly describing it. I would guess that neuron activations function like these vibes or intuitions, rather than written rules.

I very much liked the parts discussing circuit ecology. This idea sounds very promising for alignment to me.