Comment by NaiveBayesian
20 hours ago
Mixture of Experts is already used by pretty much all modern LLMs to address exactly this phenomenon.
Hopefully, future models can be trained to be even more aware of external knowledge, accessible through web search / RAG / whatever it will be then, and might not need to internalize much knowledge at all.
> future models can be trained to be even more aware of external knowledge
Then you need longer contexts, which is proving to a much more stubborn problem than general knowledge compression.
Note that MoE is a sparsification mechanism and doesn’t actually have to do with expertise or what would commonly be considered areas of expertise.