Comment by SergeAx

With all the ChatGPT boom and startups built on top o their API hassle, OpenAI receives troves of data to train and fine tune their models on. GPT 3 was trained on only 17 gigabytes of data, and GPT 4 is not far away with 45 gigabytes. On the other side, Alpaca or Vicuna was fine tuned from LLaMA using only megabytes, if not hundreds of kilobytes of training data. I believe it is much more feasible path to significantly improve current generation LLMs.