Comment by charcircuit
9 hours ago
Scale is all you need.
That's why you see other attention free large language models work. But in a world with constraints on time and compute the architecture does matter because scaling does have a cost (which is dropping)
Pretty much. See “Scaling Laws for Neural Language Models”, a must read IMO https://arxiv.org/html/2001.08361v1
Compute, data, and parameter count are the three logarithmic values that correspond to model performance.
So if you find ways of improving training efficiency (the simplest conceptually is probably local/global sliding window attention like Gemma; the latest DeepSeek and Kimis have quite fancy ones), you will be able to have more effective compute, and train smarter models while keeping data and param count the same.
If you find this fascinating and want to dive more, I encourage checking out nanochat (solid GPT-2 repro; minimal LoC); and trying to make architectural tweaks with a coding agent.
“Learning through experimentation” has never been easier or more fun in my opinion. If you have a GPU, even an older 8GB card (or a few bucks for Runpod), you can pretrain your own small language model and start experimenting / tweaking it.
Yes if you took a transformer network and added connections until it became a full network, then it would still work (?) but it would be a lot more expensive.
(?) We actually don't know that but it is very likely.
In Qwen 3.5, 75% of the layers are linear attention. I think the theory is that linear attention is usually enough, but some regular attention layers help to handle more sophisticated long-range relationships.
[flagged]
Makes me wonder if the brain just does it with brute force.
The neocortex would be akin to something like a 100T neural network, and that’s with the maybe overly reductionist assumption that only synapses are doing anything when there’s a lot of biology happening in there. Every cell is filled with its own computation and state. It could be closer to a 10E model if all that is in play.
The neocortex is nowhere near the whole brain.
Architecturally the brain could be described as a slow but ultra-wide machine with colocated memory and thus an effectively enormous memory bandwidth. That lets it evaluate its whole parameter set continuously.
I keep thinking the brain must be algorithmically doing something super clever but maybe not. Maybe old mother nature just found the “bitter lesson.” Maybe it’s algorithmically inefficient and messy but there’s just a whole lot of it.
[flagged]