Comment by gjm11
1 day ago
A convolutional neural network is really somewhat like a visual cortex. Obviously AlphaZero doesn't literally have a visual cortex -- actual literal visual cortices are features of actual literal brains made out of meat -- but it definitely has something that does something akin to visual processing, in a way that LLMs don't. Or at least they don't on the face of it; maybe well trained large enough LLMs have effectively implemented something kinda-visual-cortex-like on top of the transformer architecture.
(I bet there are people at all the big AI labs working on ways to incorporate something more CNN-like into LLMs somehow.)
I believe the architecture for convolution neural networks were directly inspired by how vision works and some of the core design choices map onto real features of the visual cortex.
That's fair. Two things that are heavily underrated are architecture and encoders