Comment by londons_explore
21 hours ago
Whilst processing in memory is clearly the future, I am unconvinced by this implementation.
Matrix multiplication involves getting every entry of the input and output matrices to be at the same multiplier at the same time. (Ie. N^2).
To do that, a lot of data movement needs to happen. Movement is the main thing - the multiplication and addition is a sideshow as far as energy and silicon space is concerned. You need a 'around the chip' ring shift register to pass every element of one matrix past every element of the other.
"Movement is the main thing" is precisely why pursuing compute-in-RAM makes some sort of sense to begin with. But DRAM fabrication processes are quite specialized and do not perform well with pure compute logic. The overall profile of this thing will arguably be similar to a rather weak NPU, though with much better memory bandwidth - one key limitation, as with NPUs, will be the bespoke programming model and lack of support for the latest compressed/quantized number formats, which heavily limits the usefulness of being able to access memory directly. GPUs, even weak iGPUs, can dequantize/pad parameters on the fly which adds a lot of flexibility - and expose standard, well understood compute capabilities via CUDA, Metal or Vulkan. This is not quite comparable unfortunately.
I was under the impression they're doing a chiplet thing to get heterogenous processes for that reason.
If what we want to do with this is make cheap QKV sweeps, then "a weak NPU with a lot of mem bandwidth" seems good enough? Exactly the tool for that job, and nothing else.
Also spares us the trouble of dealing with weights. By the time we're in QKV realm, the weights have already weighted.
Given how important matrix multiplication with a huge number of fixed parameters is becoming, there is an enormous incentive to design much more efficient architectures where this very simple compute is colocated with memory. Inference cost would come down a lot.
With the size of these matrices I don't think they are even meaningfully colocated with themselves in memory. You'll end up with some dataflow TPU architecture anyway because you'll have to stream the second matrix to multiply against.
Turning RAM into an active coprocessor does feel like an awkward model for that though.
Intel's Advanced Matrix Extensions (AMX) and the dedicated tile registers, which are basically systolic arrays IIUC, seem like a better fit.
But Samsung doesn't have that option so they're playing the card they have.
I think useful flops per dollar will outweigh any "awkwardness" factor.
Elegant designs don't win anymore.
Exactly where I see this going as well. Sure, a smartphone might be a nice place to introduce such tech. But matrix multiplication is literally where all non-labour jobs are going - this is the bedrock for efficient (time, energy) machine learning and inference.
If AI really is going to eat all our jobs, then matrix multiplication in memory is almost a requirement.
People have been calling processing-in-memory "the future" since at least the 1980s. No one has been able to reduce the concept to a useful implementation but there is a long history of failed attempts.
At this point processing-in-memory has taken on the aura of fusion power.
OLED also has a deep history with a pretty famous opinions that it is impossible and a waste of time and money for companies to invest in the development. So does AI. From inception to product it is normal for decades to pass.
Compute needs change. AI needs are pretty unique in terms of scale and type of compute requirements to anything else so far.
> Whilst processing in memory is clearly the future
How clear is that? The idea has been around for about 60 years, and many attempts made by people who thought the same thing. Maybe this time it'll be the future.
Because now the usecase fits the needs.
Maybe. The use cases have always been around FP arithmetic over arrays, because that's what is easy to parallelize. I saw a standalone systolic array box attached to a MicroVAX circa 1990. Lots of PIM approaches in the mid-90s, too, but mostly what survived from that era are shared memory NUMA multiprocessors and using GPUs for general purpose computing.
Not if you have duplicates of rows on the first matrix, which can be done very efficiently if you build specialized hardware. Then its all just forward in parallel.
you're absolutely correct that pim without a real discussion about how that works in a broader communications context is kind of useless.
what I find strange is the adoption of a standard synchronous dram interface. that's a horrible left over piece of architecture that severely constrains the applicability of this device. control flow on the dram side can't initiate any transactions on its own, or respond after work has been done - its like usb, except with a hard limit on the response.
that severely limits the utility of the in-memory processors to doing things like encryption and compression - but even then those impose delays that effect the consistency model across that interface.