Comment by hypercube33

14 hours ago

Doesn't this depend on the model and quant as well vs. the hardware and its capability? Plus memory bandwidth, amount, and layout.

My Strix Halo now can run some models through the NPU and its roughly as fast as its GPU+CPU (on some of the models - some its slower)