Comment by zozbot234

21 hours ago

"Movement is the main thing" is precisely why pursuing compute-in-RAM makes some sort of sense to begin with. But DRAM fabrication processes are quite specialized and do not perform well with pure compute logic. The overall profile of this thing will arguably be similar to a rather weak NPU, though with much better memory bandwidth - one key limitation, as with NPUs, will be the bespoke programming model and lack of support for the latest compressed/quantized number formats, which heavily limits the usefulness of being able to access memory directly. GPUs, even weak iGPUs, can dequantize/pad parameters on the fly which adds a lot of flexibility - and expose standard, well understood compute capabilities via CUDA, Metal or Vulkan. This is not quite comparable unfortunately.

I was under the impression they're doing a chiplet thing to get heterogenous processes for that reason.

If what we want to do with this is make cheap QKV sweeps, then "a weak NPU with a lot of mem bandwidth" seems good enough? Exactly the tool for that job, and nothing else.

Also spares us the trouble of dealing with weights. By the time we're in QKV realm, the weights have already weighted.