Comment by magicalhippo
17 hours ago
And a 5090 has 1.79 TB/s.
So yeah on its own not very impressive.
The large amount of memory means you can serve larger models, or less quantized, but that also means they're going to be slower due to the low bandwidth.
https://www.techpowerup.com/gpu-specs/geforce-rtx-5090.c4216
I'm pretty sure the D270 is meant to be a competitor to the RTX Pro 6000, which also has 1.79TB/s memory bandwidth like the RTX 5090. And it will potentially be a competitor (depending on what the MSRP is) to Nvidia GB10 based systems for 'small' scale AI development. It's not meant to run frontier-size models at the speed you'd expect from something like a cluster of H200 cards.
D270: 1.5TB/s memory bandwidth, 480GB memory capacity, 350W power draw
RTX Pro 6000: 1.79TB/s memory bandwidth, 96GB memory capacity, 600W power draw
Slightly less memory bandwidth, but 5x memory capacity and nearly half the power draw.
Sure, but if you use that memory for the weights of a single model, it'll be ~5x slower.