Comment by daymanstep
2 days ago
Is this meaningful? Aren't frontier models MoE anyway? If you issue a StarCraft query they will probably just route to their StarCraft optimized model.
2 days ago
Is this meaningful? Aren't frontier models MoE anyway? If you issue a StarCraft query they will probably just route to their StarCraft optimized model.
MoE is a misleading term, there is no "starcraft expert". There's not even a python expert. MoE just means a subset of blocks of weights are activated per query, nothing more. In so far as they might have legible functions they're like "balancing parens"
MoE doesn't work that way. It's not about increasing performance of a task, but speeding it up. To get a faster model, you need to make it smaller. If you make it smaller, the less it can remember. MoE tries to get faster responses without reducing the capabilities by running a smaller part of the model, and constantly swapping the parts used, so that the whole model gets to influence the output collectively, but each token is created by a smaller set of weights.
Anyone else feels like they can tell when the underlying model is MoE? Dense models are more concise in their answers. MoE seems to stall for time, writing vaugely until a right set of weights gets picked and commits to an answer. MoE also backtracks a whole lot more.