Comment by bonoboTP

Also note that MoE is a decades old term, predating deep learning. It's not supposed to be interpreted literally.