Comment by eru

A lot of what humans do is combining old ideas.

And an LLM could in theory also stumble upon entirely new ideas: there's randomness in how they generate their reasoning and answers after all.

I suspect that we are seeing a lot of advances coming from the combination of existing but somewhat obscure knowledge coming from LLMs at the moment, because LLMs are really good at this. At least compared to humans.

Even before our AI friends became good, they were already known for having read approximately every paper and every textbook published in any language. You only need to increase intelligence a fairly small amount from there to get to something like the 'convex hull' of human knowledge.

Compare https://slatestarcodex.com/2016/11/17/the-alzheimer-photo/

The gist is that basically whenever anyone comes up with a new method you get a big burst of activity of picking up all the now lower hanging fruit, that was previously out of reach.