Comment by cmrdporcupine
1 day ago
Curious how this compares -- overall -- to the RK3588 devices that I have a few of.
People have made the NPU on that thing do LLMs, and sounds like around the same level (max 3Bish params, 5-6 tok/s last time I tried).
In terms of raw CPU performance, sounds slower?
But maybe has more cores?
Ouch the memory bandwidth sounds really bad.
I don't know what kind of code sysbench is using, but I get far better with a very simple `memcpy()` loop:
See https://news.ycombinator.com/item?id=48523343