Comment by mabbo
3 years ago
They need a few algorithmic improvements first, imho. GPT4 is noticeably slower than GPT3.5 and apparently costs a lot more to use, implying some serious compute costs.
They could train it with more data in the hopes of getting another big leap there, but what data is left? They've fed it everything it seems.
So what's left is getting the runtime reduced in terms of the model size. Hire some brilliant minds to turn an N-squared into an N-log-N (or something to that effect).
Maybe GPT4 has some ideas.