Comment by darth_avocado
3 months ago
More than that, I think people overestimate how much AI will progress as you throw more compute at it. It’s the “9 women can’t deliver a baby in a month” equivalent of AI. Additional compute won’t magically give you AGI.
How do you know though? People thought ANNs and perceptrons were basically useless until pretty much around the time of AlexNet and while there were some architectural improvements (CNNs, transformers), the biggest thing that changed was that we simply threw more compute at it.
It's definitely too early to declare that more compute won't make a difference.
It was data. Alexnet worked because imagenet existed. It took less than a year between the dataset becoming public and the first version of Alexnet being trained.
The Allegorical Agent Aeon
Maybe not AGI, but if you look at the differences between, say, GPT-2 and GPT 5.5, it's remarkable how well it works to mostly just throw scale at the problem.
The difference is a lot more than just throwing scale at it, pretty much everything useful comes from an evolving landscape of post-training techniques.
Of course, param count and context length are also important because they increase the model's overall fidelity, but a base model without SFT, RHLF etc is effectively useless.
Correct. That is what I was trying to hint at. Yes, massive compute is needed to train ai, but it isn’t the only thing. A lot of research and experimentation goes into moving the marker just a little bit. Innovation can’t be forced into weekly sprints, it takes its own time.
6 replies →
I was under the impression that the deltas between versions were shrinking- i.e. gpt 4 -> 5 was much less impactful than 3 -> 4 or 2 -> 3. If the growth is getting diminishing returns, I can't say I'm optimistic without finding a drastically different approach.
GPT 4/5 etc. are just marketing names though.
They already tried that with GPT-4 and GPT-4.5
They were allegedly massive but the cost and returns were not worth it.
Not really. I didn't use GPT-2, but I don't think there was much difference between we got what out of GPT-3.5 and 5.5. It's still an unreliable tool which will go off the rails the second you aren't watching it like a hawk. It still has zero intelligence or ability to reason as to why its output is flawed. Just throwing more compute hasn't gotten us anything worth using.