How useful would that actually be for local inference, when the bottleneck is, essentially, always memory size? I think the sweet spot is somewhere around 64 to 128 GB. That ought to be enough for almost all uses, like Qwen3.8-Flash-Next at a decent quant.
Oh interesting, any more info on "how"?
With a three point plan, I suspect: memory bandwidth, memory bandwidth, and more memory bandwidth.
How useful would that actually be for local inference, when the bottleneck is, essentially, always memory size? I think the sweet spot is somewhere around 64 to 128 GB. That ought to be enough for almost all uses, like Qwen3.8-Flash-Next at a decent quant.
3 replies →
Plus with more low precision compute… to help with the memory bandwidth!
More ASIC Mat-Mul too.