Comment by swiftcoder
3 hours ago
> Not sure if you misunderstand what GPUs we're talking about, one RTX Pro 6000 has 96GB of VRAM.
Right, but the top option here is a pair of RTX Pro 6000s, hence 192 GB of VRAM in total. Should be enough for a 128GB model plus context, cache, etc.
Yeah, that's why the "any model over about 128GB of weights isn't going to fit on the GPUs" part doesn't make sense, you'll easily be able to run weights in that weight class on two of them. Or did I misunderstand what "on the GPUs" you meant?
Right, easily run 128-160 GB models, yes. Not easily run models much larger than that. Anything that doesn't fit in the 192 GB (including context, cache, etc) is going to have to be sparse/MoE, and the rather anaemic PCIE bandwidth is going to hurt.
By comparison, the 5x Mac cluster should be able to run a dense ~800GB-1TB model without a drastic slowdown.