Comment by observationist
3 days ago
The idea that tokenization is what they're for is absurd - you're talking a tenth of a thousandth of a millionth of a percent of efficiency gain in real world usage, if that, and only if someone bothers to implement it in software that actually gets used.
NPUs are racing stripes, nothing more. No killer features or utility, they probably just had stock and a good deal they could market and tap into the AI wave with.
NPUs aren't meant for LLMs. There are a lot more neural net tech out there than LLMs.
> NPUs aren't meant for LLMs. There are a lot more neural net tech out there than LLMs.
OK, but where can I find demo applications of these that will blow my mind (and make me want to buy a PC with an NPU)?
Apple demonstrates this far better. I use their Photos app to manage my family pictures. I can search my images by visible text, by facial recognition, or by description (vector search). It automatically composes "memories" which are little thematic video slideshows. The FaceTime camera automatically keeps my head in frame, and does software panning and zooming as necessary. Automatic caption generation.
This is normal, standard, expected behavior, not blow your midn stuff. Everyone is used to having it. But where do you think the computation is happening? There's a reason that a few years back Apple pushed to deprecate older systems that didn't have the NPU.
3 replies →
Best NPU app so far is Trex for Mac.
I think they were talking about prefill, which is typically compute-bound.