Comment by dougmwne

Based on published research from Google and Meta it is somewhat known how much more capability is possible with the current approach, but it would require an extreme increase in compute and training set to achieve it. There are diminishing returns, but the returns appear to continue for a good while, even without any new model architecture discoveries. Right now the expense will likely mean that progress will be limited to the pace of Moore’s law.

In terms of what this improvement would actually look like in terms of real world, emergent capabilities, no one knows.