Comment by syntaxing
19 hours ago
I use it with a strix halo server. 35B runs stupidly fast. 27B is about 700 TPS prefill and 30 TPS token generation. Which interestedly is about what Kimi K3 gives me depending on provider.
19 hours ago
I use it with a strix halo server. 35B runs stupidly fast. 27B is about 700 TPS prefill and 30 TPS token generation. Which interestedly is about what Kimi K3 gives me depending on provider.
what hardware do you use or recommend for this? never heard of it until today.
Strix Halo is the unified memory platform from AMD. Similar to the DGX Spark from NVIDIA or the M series Macs.
I personally have the Framework Desktop, but there's also systems from other brands like Bosgame
You can also get it in a laptop form factor that feels like a MBP with a nicer keyboard if you get an HP Zbook G1A!
Huge fan of that thing, it's th e Linux MBP I've always wanted.
1 reply →
I have a framework desktop, but depending on your need, DGX spark might be better. The prefill and NVFP4 is a significant advantage. But framework desktop is a better general computer. I expect to be able to use it for years to come. Where as DGX Spark you’re at the mercy of NVIDIA BSP.
RTX 4060 and above. Ideally RTX 50 Series, because you can run NVFP4-quantized GGUFs that give you better prefill AND better quality.