Comment by com2kid
19 days ago
If Moore's Law had fully kicked over twice more we'd all have 64GB GPUs, enthusiasts would have 2x64GB, and data center build outs wouldn't be needed.
Eventually GPU memory is going to creep up and local models will powerful enough.
I agree. I also think we have only hit the surface of model efficiencies.
Apple's M3 Ultra with RAM up to 512GB shared directly across CPU/GPU/NPUs is a great example of an architecture already optimized for local models. I expect Apple will start offering larger RAM sizes for other form factors too.
And prices for RAM will drop eventually, because of the extreme demand for RAM with higher densities.
It reminds me of the huge infra investments in Sun and Cisco during the first .com boom, and then 5-10 years later those fancy Sun boxes were out performed by Grandma's Windows XP box.