All you have to look at is ASIC miners. Once they had them, they were 10x faster than GPUs easily and made GPUs useless for those algos. Something very similar can happen soon.
The fundamentals are different. Bitcoin mining is not intrinsically suited to acceleration on a GPU. It is a not-very-wide serial integer operation.
AI inference on the other hand is basically just very large floating point tensor matrix multiplication. What does an ASIC for matmul look like? A GPU.
No, GPUs are not a particularly ideal architecture for AI inference, it's just inference needs way more memory bandwidth than a general purpose CPU's memory hierarchy can handle.
> What does an ASIC for matmul look like?
A systolic array, and ultimately quite different than a GPU. This is why TPUs et all are a thing.
In general with a systolic array you get a quadratic speedup. For example with a 256x256 array, it takes 256 cycles to shift operations in and out, but in doing that you accomplish 65k MACs in 512 cycles for a speedup of 128x over serial.
This was sponsored by AMD for quite a while and dropped. There had to be some talks behind closed doors that resulted in the current situation. I mean, we all know they should've and they're not dumb either.
All you have to look at is ASIC miners. Once they had them, they were 10x faster than GPUs easily and made GPUs useless for those algos. Something very similar can happen soon.
The fundamentals are different. Bitcoin mining is not intrinsically suited to acceleration on a GPU. It is a not-very-wide serial integer operation.
AI inference on the other hand is basically just very large floating point tensor matrix multiplication. What does an ASIC for matmul look like? A GPU.
No, GPUs are not a particularly ideal architecture for AI inference, it's just inference needs way more memory bandwidth than a general purpose CPU's memory hierarchy can handle.
> What does an ASIC for matmul look like?
A systolic array, and ultimately quite different than a GPU. This is why TPUs et all are a thing.
In general with a systolic array you get a quadratic speedup. For example with a 256x256 array, it takes 256 cycles to shift operations in and out, but in doing that you accomplish 65k MACs in 512 cycles for a speedup of 128x over serial.
2 replies →
Sorta? If that was the full story, TPU would not be a thing.
3 replies →
Asic for matmul is systolic array more or less
They don't even need to invent anything. Just put in some work to polish what already exists: https://github.com/vosen/ZLUDA
IMHO AMD should have done this a decade ago.
This was sponsored by AMD for quite a while and dropped. There had to be some talks behind closed doors that resulted in the current situation. I mean, we all know they should've and they're not dumb either.