Comment by littlestymaar

> Dedicated memory isn't the issue.

To run large MoE models it is.

> Increase DRAM on your card and your bandwidth goes down

Why would it?

> You'll note that Apple didn't just immediately resume shipping systems with 1.5TB of RAM when they revised their own system architecture. It's taken them half a decade to recoup a third of that capacity at the VRAM-level speeds they require to unify the GPU and CPU's memory

I fail to see how a unified architecture on a general purpose CPU is a good illustration when we're discussing PCIe accelerator cards. The problems they face have little in common.