Comment by po1nt
18 hours ago
I think you could do many performant things without any involvement of software. For example you could do AVX on RAM. The CPU would recognize PID RAM and offload AVX instructions to the module.
Then, by simply asking for a special memory address you could have access to registers/regions within PID RAM that serve as a result region.
Let's say you would need to run a sum over megabytes of data like for accounting. You could just ask RAM to do it and load just the result. The bandwidth would could be 8x higher and software could stay the same.
Doing scalar operations, frequent dereferencing and similar would not get much peformance benefit in many cases, as loading and accessing CPU cache is often much faster. But simple vector operations over large data could be massive.
Having accelerators on RAM like for jpeg compression, audio decoding or mass data operations could be beneficial but you would need to be careful with heat dissipation.
Personally I'm a big fan of the "in-ram accelerator" idea especially for server space. Doing fuzzy search in RAM could be massive performance improvement.
Yeah this sounds a lot like a natural evolution of SIMD for me, just cut out the middleman and put the SIMD units straight into RAM.
I can imagine some power savings for always-on display applications too. Rather than periodically waking the CPU/GPU to update the frame buffer, you can just stash small bits of periodic logic in memory (e.g. move the second hand of a clock).
I'm having a hard time imagining this SIMD replacement except for extremely narrow use cases. Are you suggesting the PIM would have a full blown IO controller and cache subsystem to fetch remote operands?
I assume PIM is only going to work well for chunky streaming over the data within that particular memory module. Something that address and operate on whole rows at once and has minimal buffering between the RAM access ports and the PIM register state.
A lot of SIMD code can be on two array operands, and I expect this PIM approach only works well if both are stored locally in the same memory "local" to the PIM and where it can efficiently interleave at the natural addresisng and access widths. Too much random access or needing "remote" data sounds like the point where PIM fails and you need the elaborate memory IO controllers and caching subsystems of CPUs sitting on top of the distributed memory modules..?
> PIM would have a full blown IO controller and cache subsystem to fetch remote operands
Don't we already have those in mainstream computing in the form of dedicated silicon in DMA controllers? Programmed input–output performance is often low throughput, high jitter and uses a lot of CPU.
1 reply →
Data movement and local operations are still bottlenecked today on memory bandwidth. Butterfly primitives, sorting/fft/1D-convolution, the whole cub library, could be ported there and have great performance wins. But the pain of programming and maintaining code using this...
> run a sum over megabytes of data like for accounting
It's been many decades since the last time somebody ran a sum over megabytes of data for accounting and though "damn that's a bottleneck I need to optimize".
> Personally I'm a big fan of the "in-ram accelerator" idea especially for server space.
The operations this model supports are so extremely limited that you would be hard pressed to find applications where it's worth it.
> Having accelerators on RAM like for jpeg compression, audio decoding or mass data operations could be beneficial but you would need to be careful with heat dissipation.
I think we are essentially reinventing SSE, AVX & friends from first principles. This is already being done. Compare the speed of libjpegturbo to a non-vectorized implementation and you'll find a 2-4x difference in throughput.
If Infiniband does this for MPI on the network and realizes Sun's "The network is the computer" dream, I believe we can do this for other parts of the hardware, as well. Not only for AI, HPC will love this idea.