Comment by _davide_

> Apple Silicon is a natural fit for tensor/matrix operations due to unified memory,

This is a common misconception. This is NOT true: memory designed for CPU performs horribly for GPU tasks. Mac/Strix Halo/NVIDIA Spark are performing horribly compared to a desktop video card with GDDR.

I spent quite a few weekends optimizing ROCM kernels for strix halo, i WISH I had GDDR instead of high-frequency CPU RAM; the bandwidth would be SO MUCH better.

I'm quantizing weights not to make computation faster, not because I'm out of memory, but because memory cannot move fast enough to be processed and it's cheaper to load quantized version, convert into bf16 compute, and discard; This happens every single time a token is generated for the whole model over and over.