← Back to context Comment by bigyabai 4 months ago Not even a cluster of Mac Pros could run a dense 5T parameter model with RDMA, to my knowledge. 4 comments bigyabai Reply zozbot234 4 months ago SOTA models are reportedly MoE, not dense. bigyabai 4 months ago A 5T MoE model is still bottlenecked by streaming weights from SSD, in addition to compute bottlenecks during prefill and decode. zozbot234 4 months ago True but a cluster built on pipeline parallelism can naturally stream from multiple SSD's in parallel. That probably makes offload somewhat more effective. And you also have RAM caching available as a natural possibility. 1 reply →
zozbot234 4 months ago SOTA models are reportedly MoE, not dense. bigyabai 4 months ago A 5T MoE model is still bottlenecked by streaming weights from SSD, in addition to compute bottlenecks during prefill and decode. zozbot234 4 months ago True but a cluster built on pipeline parallelism can naturally stream from multiple SSD's in parallel. That probably makes offload somewhat more effective. And you also have RAM caching available as a natural possibility. 1 reply →
bigyabai 4 months ago A 5T MoE model is still bottlenecked by streaming weights from SSD, in addition to compute bottlenecks during prefill and decode. zozbot234 4 months ago True but a cluster built on pipeline parallelism can naturally stream from multiple SSD's in parallel. That probably makes offload somewhat more effective. And you also have RAM caching available as a natural possibility. 1 reply →
zozbot234 4 months ago True but a cluster built on pipeline parallelism can naturally stream from multiple SSD's in parallel. That probably makes offload somewhat more effective. And you also have RAM caching available as a natural possibility. 1 reply →
SOTA models are reportedly MoE, not dense.
A 5T MoE model is still bottlenecked by streaming weights from SSD, in addition to compute bottlenecks during prefill and decode.
True but a cluster built on pipeline parallelism can naturally stream from multiple SSD's in parallel. That probably makes offload somewhat more effective. And you also have RAM caching available as a natural possibility.
1 reply →