Comment by LoganDark

Qwen seems to be going in a good direction -- hundreds of experts on their MoE models. Extremely low active-weight counts while still performing quite admirably. I look forward to models with many, many more experts, to the point where anyone with enough random access can generate hundreds or thousands of tokens per second. Because right now, 80–120t/s is pretty slow.