Comment by cgearhart
19 hours ago
What exactly is driving all these capability improvements? Are they making more data? Using existing data more effectively? Modifying architecture? Changing the training somehow to pick up slack that was previously left behind?
I’m kinda surprised that we expect the models to have infinite capacity to improve. It seems clear that we haven’t hit any fundamental limit yet, but that doesn’t mean no limit exists. If it does start improving exponentially, then either we’ve dramatically underestimated their capacity or they’ll hit that limit quite quickly, right?
It's hard to say for sure because the closed labs don't share anything about their improvements. However, if you look at DeepSeeks papers, most of the improvements come from:
* Better RL environments that themselves are generated by agents
* Architecture improvements that increase effective capability per parameter or FLOP
If you're curious about the latter I'd look into manifold constrained hyper connections, engram, deepseek compressed attention, kimi delta attention etc. At first these might seem like just speedup tricks, but several of these allow the overall capacity of the model to increase per training FLOP or parameter. This then lets pretraining & the RL environments grow & be more effective.
I’m broadly aware with some of this and specifically aware of the work on RL environments. That’s exactly my point—the models are still a bit bigger and we know that there’s lots of capacity left to improve capabilities, but it’s not getting orders of magnitude bigger like they were originally and if it’s just “more RL” then we’re consuming a finite capacity more effectively. So either we’ve exponentially underestimated the full capacity of the models at current scale or any kind of exponential improvement is going to fill the available capacity extremely fast. It remains unclear to what degree of “superintelligence” a full-capacity current model will achieve. (It’s already quite far past human intelligence in many specific capabilities.)
Articles like the OP seem to imply that there is no fundamental limit or that it’s just inconceivably far out. I’m sure that AI helps build AI as much for them as it does for me, but I’m not clear on the argument in support of infinite growth.