Comment by jmyeet
10 hours ago
The 256GB M5 Ultra Mac Studio can (AFAIK) run the latest open weight Flash models but only 2 bit quant. This by extension will be able to do 4 and that's a big difference. We're not at the point of running 1T+ parameter models locally yet but we'll be there by 2030 easily.
I personally believe that local LLMs have a huge captive market for people who cannot or should not host that data with a third-party. For example, it's an open question of whether or not using cloud LLMs by lawyers breaks attorney-client privilege. HIPAA with medical records is another one.
But I want to expound on one point that you made, which is really important. You say that this spec only has a SOTA life of ~1.5 years and that's an issue. Take that to its natural conclusion. Why isn't it an issue for hyperscalar data centers, particularly as NVidia wants to move to annual chip updates.
I believe the hardware is going to be commoditized. At that point the only thing that matters is performance-per-Watt. NVidia's R100/200 doesn't make the B100/200 obsolete but if the R series gets 50-100% PPW more and these DCs are using gigawatts of power, that's a massive difference.
What really is the effective life of AI hardware?
> Take that to its natural conclusion.
What on earth is a "natural conclusion" in this context? I'm going to assume that part of your comment was written by AI.
> Why isn't it an issue for hyperscalar data centers, particularly as NVidia wants to move to annual chip updates.
Because he's not a hyperscaler? He's not operating data centers where compute is fungible, he's buying an individual machine that he wants to use for years. He can't just keep buying a new machine and fob off the old one to someone else's less demanding workload.
Enterprise GPUs have a lot longer effective life than these consumer chips (although even the old 3090s are reasonably useful, especially with the recent driver unlock tech). I'd argue that these consumer chips have only now gotten to the memory capacity/processing capacity to make local LLM use actually reasonable.