Comment by hedgehog
10 hours ago
Separating knowledge from reasoning so you only pay for what you use is a big rationale for MoE, the big problem being MoE training has historically been hard to get right. In a dense model every single token you're paying a cost to determine whether you're now talking about the flavor of durian.
Sure, the "6b subset" can be more knowledgeable on its area than a whole 27b generalist (and more efficient), but where is the simulated Intelligence encoded? A 6b subset as or more intelligent than a 27b raises the question of how metacognition skills are stored.
MOEs are built by training a second "router" model to identify which parts matter inside the dense model.
Think of MOE as ignoring noise, rather than a more efficient encoding of the data we teach it which ones can be ignored. Turning down the noise actually sharpens the results sometimes.
Modern LLM's are wildly inefficient.