Comment by nunez
2 hours ago
They've been investing in NPUs for a long long time. Since the first iPad Pro at least (2018). Apple Silicon is a natural fit for tensor/matrix operations due to unified memory, so if anything them going "all in" on that has been a nice side effect.
> Apple Silicon is a natural fit for tensor/matrix operations due to unified memory,
This is a common misconception. This is NOT true: memory designed for CPU performs horribly for GPU tasks. Mac/Strix Halo/NVIDIA Spark are performing horribly compared to a desktop video card with GDDR.
I spent quite a few weekends optimizing ROCM kernels for strix halo, i WISH I had GDDR instead of high-frequency CPU RAM; the bandwidth would be SO MUCH better.
I'm quantizing weights not to make computation faster, not because I'm out of memory, but because memory cannot move fast enough to be processed and it's cheaper to load quantized version, convert into bf16 compute, and discard; This happens every single time a token is generated for the whole model over and over.