Comment by jumploops
7 hours ago
My biggest problem with running local LLMs on my M4 Max/128GB RAM is the prefill latency.
I've since acquired two DGX Sparks, and it feels so much snappier.
7 hours ago
My biggest problem with running local LLMs on my M4 Max/128GB RAM is the prefill latency.
I've since acquired two DGX Sparks, and it feels so much snappier.
Would you mind sharing your local Mac setup and which models you currently use and whether it’s GGUF or MLX? I’ve the hardware same specs.
Primarily used ds4[0] by antirez
[0]https://github.com/antirez/ds4
Thanks, that’s what I’m currently using too.
m5 max really fixed pp with the better matmul support, im sure the m5 ultra will be even crazier
the sparks have much slower memory bandwidth is the trade off
I believe the dgx spark is still twice as fast at prefill as the m5 max, but the ultra should get closer to parity.
Another benefit of the 2x spark setup is that you can parallelize to ~6 streams pretty efficiently.
All depends on the workflows you’re using it for.
I’m quite excited for the M7 class machines.