Comment by naasking

> I would expect the performance on a 768B parameter model spread across 4 to be significantly lower even though its a mixture of experts.

To fit that model you'd need multiple machines which lets you run tensor parallelism for additional speedups, roughly X speedup for X machines. Also you can't naively compare tps across models, and it would certainly run a nvfp4 quant or similar.