Comment by imtringued

For GEMV you lose nothing.

The reason is quite simple. You can split the matrix along both dimensions so you just tile it into 64x64 or whatever fits into the bank and just fill it up. The biggest problem is load balancing the tiles across all banks for maximum parallelism.

For GEMM I believe there is no point in doing PIM, you are better off with a GPU or NPU.