Comment by kmike84
8 hours ago
It’s making ddr3 relevant again as well. Bought 256GB DDR3 ECC for ~200 usd a couple weeks ago for my ai server (wip).
These old workstation platforms seem to be perfectly fine for hosting GPUs for local llms.
8 hours ago
It’s making ddr3 relevant again as well. Bought 256GB DDR3 ECC for ~200 usd a couple weeks ago for my ai server (wip).
These old workstation platforms seem to be perfectly fine for hosting GPUs for local llms.
256GB means you are going to be using system RAM instead of completely offloading to GPU, I assume? If so, you should make sure your CPU has AVX2.
I think it'd be mostly for engrams. It seems LLMs are moving in this direction, with 50GB in qwen 3.8 flash, and 200GB in deepseek 4.1 flash.
My $20 xeons 2696v2 and 2670 are tool old for real offloading, and sata ssd drives are likely a bit slow for ssd streaming :)
"hosting GPUs for local llms" meaning pci-express bandwidth and memory bandwidth are not issues for them, and CPU features probably also don't make a difference.