Comment by DoctorOetker

more interesting would be using some kind of FPGA to logic glue each RAM socket interface bitplane to a hard drive (so a collection of hard drives with ridiculous collective bandwidth). Perhaps a single RAM socket contains actual RAM and the linux kernel would have to be modified to only use the real RAM memory region for OS and inference software, with the inference software rewritten to stream LLM weights deterministically from the hijacked RAM slot physical memory regions. Obviously the FPGA can't truly achieve the CAS latencies over the HDD (unless the HDD firmware was rewritten so it can predict the next deterministic token sufficiently in advance to cache the result and stream it just in time to FPGA then "RAM" socket...) but even if the HDD firmware can't be reprogrammed for some reason, the FPGA knows what memory address will be deterministically fetched next, so it can make the requests to the parallel array of HDD's ahead of time.