Comment by tim333
3 months ago
The words 'lead to' there cover a lot. I don't think we'll get AGI just by giving more compute to the models but modifying the algorithms could cover a lot of things.
Like at the moment I think during training new data changes all the model weights which is very compute intensive and makes it hard to learn new things after training. The human brain seems to do it in a more compartmentalised way - learning about a new animal say does not rewrite the neurons for playing chess or speaking French for example. You could maybe modify the LLM algo along those lines without throwing it away entirely.
The need for new data seems like it has outpaced the rate at which real data is being generated. And most of the new data is llm slop.
So you might improve algorithms (by doing matrix multiplications in a different order.... it's always matrix multiplications) but you'll be feeding them junk.
So they need ever increasing amounts of data but they are also the cause of the ever increasing shortage of good data. They have dug their own grave.