Comment by danmaz74
20 hours ago
Given how important matrix multiplication with a huge number of fixed parameters is becoming, there is an enormous incentive to design much more efficient architectures where this very simple compute is colocated with memory. Inference cost would come down a lot.
With the size of these matrices I don't think they are even meaningfully colocated with themselves in memory. You'll end up with some dataflow TPU architecture anyway because you'll have to stream the second matrix to multiply against.
Turning RAM into an active coprocessor does feel like an awkward model for that though.
Intel's Advanced Matrix Extensions (AMX) and the dedicated tile registers, which are basically systolic arrays IIUC, seem like a better fit.
But Samsung doesn't have that option so they're playing the card they have.
I think useful flops per dollar will outweigh any "awkwardness" factor.
Elegant designs don't win anymore.
Exactly where I see this going as well. Sure, a smartphone might be a nice place to introduce such tech. But matrix multiplication is literally where all non-labour jobs are going - this is the bedrock for efficient (time, energy) machine learning and inference.
If AI really is going to eat all our jobs, then matrix multiplication in memory is almost a requirement.