Comment by reliabilityguy
21 hours ago
> You could run MACs directly in RAM
Sure, MACs are nice. However, unless there other, PIM-specific/optimal, algorithms, regular matrix multiplication algorithms like tiling-based won’t work here I think — how would the tile be shared? By doing read/write all the time?
Attention calculations aren't shared across more than one vector during next token prediction (thinking and writing) which this sounds almost perfect for. Per attention layer, for deepseek at 1M context, you want to broadcast a single 1KB vector to 4GB of dot products, and map reduce a 1KB vector back.
How exactly the map-reduce will happen though? Won’t you need to do it host-side, or make a lot of reads and writes?
Also, doesn’t it mean that you forgo batching?
Map-reduce is implemented as a rolling calc, see: online softmax in FlashAttention kernels.
2 replies →