Comment by booleandilemma
3 years ago
Is it known how much of an improvement GPT-5 would bring? Can we predict how much better it would be compared to GPT-4?
3 years ago
Is it known how much of an improvement GPT-5 would bring? Can we predict how much better it would be compared to GPT-4?
Based on published research from Google and Meta it is somewhat known how much more capability is possible with the current approach, but it would require an extreme increase in compute and training set to achieve it. There are diminishing returns, but the returns appear to continue for a good while, even without any new model architecture discoveries. Right now the expense will likely mean that progress will be limited to the pace of Moore’s law.
In terms of what this improvement would actually look like in terms of real world, emergent capabilities, no one knows.
No, but llama's training architecture was designed around studying the curve of output quality vs training data size so we do have companies looking into this.