Comment by Eisenstein
8 hours ago
256GB means you are going to be using system RAM instead of completely offloading to GPU, I assume? If so, you should make sure your CPU has AVX2.
8 hours ago
256GB means you are going to be using system RAM instead of completely offloading to GPU, I assume? If so, you should make sure your CPU has AVX2.
I think it'd be mostly for engrams. It seems LLMs are moving in this direction, with 50GB in qwen 3.8 flash, and 200GB in deepseek 4.1 flash.
My $20 xeons 2696v2 and 2670 are tool old for real offloading, and sata ssd drives are likely a bit slow for ssd streaming :)
"hosting GPUs for local llms" meaning pci-express bandwidth and memory bandwidth are not issues for them, and CPU features probably also don't make a difference.