Comment by reliabilityguy
12 hours ago
How exactly the map-reduce will happen though? Won’t you need to do it host-side, or make a lot of reads and writes?
Also, doesn’t it mean that you forgo batching?
12 hours ago
How exactly the map-reduce will happen though? Won’t you need to do it host-side, or make a lot of reads and writes?
Also, doesn’t it mean that you forgo batching?
Map-reduce is implemented as a rolling calc, see: online softmax in FlashAttention kernels.
Rollie calculation, like the online softmax in FA, implies a centralized computing unit that does the compute and stores the intermediate results in its registers. With PIM you have no centralized compute unit, you have a bunch of memory, and a bunch of MACs all over the place.
How would you do map-reduce across multiple DIMMs w/o extra reads/writes?
PIM implies some sort of distributed compute, which can work for some cases, but I am not sure LLMs are one of them.
Re-arrange x+(y+(z+w)) into (x+y)+(z+w) to accomplish a distributed sum. Shared bus, interconnect fabric. This is a supercomputer on a chip.