Comment by xlayn
12 hours ago
I want to propose a called AI Index, if you can do say 1000t/s on processing and 50t/s on generation on a min 25B model you can call your device AI9 and you have AI7 and AI5 and AI3 with 3 being 100t/s processing and 10t/s generation on a min 10B model. Values above are on a 100k context size minimum. OFFLINE.
So any effing claim of being an AI device doesn't end on the demo of best buy laptop where you remove a dog from a photo....
Or that you have "better noise removal" or "you look better on the cam" because that's called noise cancelling and instagram filter and it's being done for ages...
Doesn't this depend on the model and quant as well vs. the hardware and its capability? Plus memory bandwidth, amount, and layout.
My Strix Halo now can run some models through the NPU and its roughly as fast as its GPU+CPU (on some of the models - some its slower)
Can't you just look at memory bandwidth for generation?
You also need to look at memory size (to store the model). And processing can still be the bottleneck.
It’s more nuanced than that, but yes that’s the biggest factor.