From a quick google search a DGX spark seems to decode Llama 3.1 70B (FP8) at 2 tokens per second. I would expect the performance on a 768B parameter model spread across 4 to be significantly lower even though its a mixture of experts.
For real work anything below 60 tokens per second is essentially unusable.
That's not taking into account the prompt filling, Llama 3.1. 70b on DGX spark runs at about 800 tps running at that speed prompt filling a 512k context takes like 11 minutes.
> I would expect the performance on a 768B parameter model spread across 4 to be significantly lower even though its a mixture of experts.
To fit that model you'd need multiple machines which lets you run tensor parallelism for additional speedups, roughly X speedup for X machines. Also you can't naively compare tps across models, and it would certainly run a nvfp4 quant or similar.
From a quick google search a DGX spark seems to decode Llama 3.1 70B (FP8) at 2 tokens per second. I would expect the performance on a 768B parameter model spread across 4 to be significantly lower even though its a mixture of experts.
For real work anything below 60 tokens per second is essentially unusable. That's not taking into account the prompt filling, Llama 3.1. 70b on DGX spark runs at about 800 tps running at that speed prompt filling a 512k context takes like 11 minutes.
> I would expect the performance on a 768B parameter model spread across 4 to be significantly lower even though its a mixture of experts.
To fit that model you'd need multiple machines which lets you run tensor parallelism for additional speedups, roughly X speedup for X machines. Also you can't naively compare tps across models, and it would certainly run a nvfp4 quant or similar.
It would run so slow it would be functionally unusable or at such a low quant that it wouldn't be useful for serious work.
nvfp4 is pretty good, and with 4 machines you get a roughly 4x speedup using tensor parallelism.