Comment by swiftcoder

2 hours ago

Right, easily run 128-160 GB models, yes. Not easily run models much larger than that. Anything that doesn't fit in the 192 GB (including context, cache, etc) is going to have to be sparse/MoE, and the rather anaemic PCIE bandwidth is going to hurt.

By comparison, the 5x Mac cluster should be able to run a dense ~800GB-1TB model without a drastic slowdown.