Comment by Mars008

One problem is in datasets which use RAG. Training foundational model requires a lot of samples, and there aren't many. The only option is to use other models to generate, so called distillation.

BTW, RAG is similar to web search. Models can do it. Web server for RAG can be implemented.