Comment by lemonish97
3 days ago
I wonder when we begin to see the dividends of all the NPU PCs come into play. AMD have been doing some good work with their NPU/iGPU hybrid inference kernels. If these larger models could be scaled down to run on NPUs, you'd see much better power advantages, compared to running them on the CPU.
> I wonder when we begin to see the dividends of all the NPU PCs come into play.
A few months ago I used Whisper from OpenAI, an automatic speech recognition system released in 2002, on my modern 20-core Intel CPU to convert audio from a video file to text. It worked fine. Took a while and the machine got hot and the fans kicked in. I then found the Intel's optimized version of whisper that used NPU. It required a lot more steps to get working, but in the end it did work and was about 6x faster. And the machine remained cool and silent in the process. Since then I have become a fan of the NPUs. They are not NVIDIA GeForce RTX 5090, but they are significantly better than a modern CPU.
You can already run some models on the NPUs in the Rockchip RK3588 SBCs which are pretty abundant.
A claude 4.6 they are most certainly not, but if you get through the janky AF software ecosystem they can run small LLMs reasonably well with basically zero CPU/GPU usage
Are the NPUs really that powerful?
I was under the impression that they were primarily designed for low power use.
They seem to be getting better or more powerful. The newer Intel Panther lakes and AMD Ryzen are over 50 TOPS now, IIRC