Comment by Tuna-Fish

The Taalas architecture makes loading weights free, but they basically has to pay the same silicon for every weight, whether it's used or not. MoE models are more efficient than dense models per weight you load, but less efficient per weight you have to store.

MoE models are the path to local models with traditional system architectures, but they are antithetical to what Taalas was doing. If you spent all the money to etch 125B weights into silicon, you'd want to activate them all for each token, instead of only touching 6B. You cannot match the 125B sparse model with a 27B dense one, but you might be able to match it with a 60B or so one.