Comment by softwarewright

The value proposition I am pursuing is not faster speed, but increased throughput (and possibly reduced kWh).

I'm trying to understand the benefits of streaming expert weights through hardware that offloads the math and avoids storing all of the weights in RAM at once.

But that's not the only thing that can be streamed and offloaded. '

Finally, I'm trying to come up with approaches to reuse old hardware, old GPUs, old RAM instead of paying today's prices for GPU VRAM or unified memory. Even if I do not end up showing any particular FPGA benefit, I might be able to better run very large models on systems without GPUs or without unified RAM.