Comment by imtringued

I'm not an expert, but MoE models perform better at continuous learning, because they are less prone to catastrophic forgetting.