It can run on one strix halo machine, which is what I have. For everything else, there’s a vibrant community brute forcing more efficient engines and quantizations. There’s a minimum bar right now, so it depends on your hardware.
I bought my box for $3k last year because I hadn't updated my PC since the first threadripper 1950x, and the Strix Halo is the only x86 with unified memory that's optimized for the kind of CPU intensive stuff I do, and they finally have proper AVX512 too. It was the perfect upgrade for me (except for lack of PCIe 5 & ECC), which coincidentally happened to be perfectly positioned for local inferrence.
and I liked it so much (and saw that the RAM drought was going to last for at least a few years) so I bought a second one to use as a general server for myself and my family.
It can run on one strix halo machine, which is what I have. For everything else, there’s a vibrant community brute forcing more efficient engines and quantizations. There’s a minimum bar right now, so it depends on your hardware.
you have 5k$ of ram in a strix halo?
I bought my box for $3k last year because I hadn't updated my PC since the first threadripper 1950x, and the Strix Halo is the only x86 with unified memory that's optimized for the kind of CPU intensive stuff I do, and they finally have proper AVX512 too. It was the perfect upgrade for me (except for lack of PCIe 5 & ECC), which coincidentally happened to be perfectly positioned for local inferrence.
and I liked it so much (and saw that the RAM drought was going to last for at least a few years) so I bought a second one to use as a general server for myself and my family.