Comment by auspiv
21 hours ago
This is a very low number for a server GPU. My V100 from 2017 is 0.9 TB/s.
A single B300 has almost 8 TB/s.
AMD MI300x from 2023 is 5.3 TB/s
Edit: my bad you are saying that this is an impressive number precisely because it is more or less normal RAM, not VRAM
VRAM isn't a type of memory, it's just RAM used for the GPU. So this LPDDR5 is VRAM, just like it can be with my Strix Halo machine.
That's only partially true - the GDDRX you'd find on say a 3090 is physically very different than what you'd find on a strix halo or mac or vanilla ram. e.g. runs 4 voltage levels not 2 so that it can transmit twice the info per cycle
Well, what they don't tell you about GDDR6X is that it's 4 voltage levels but it never goes from min to max or vice versa in a single transition (PAM4 MTA). It transfers 1.75 bits per cycle.
And then GDDR7 backed off to PAM3 and sends 1.5 bits per cycle.
There's a good chance [LP]DDR will use the same techniques once it gets into the same speed range.
Yes the cells are pretty much the same as they would be with hbm as well all the magic happens at the bus and the signaling, there is both a reason hbm and gddr have higher bandwidths and tradeoffs.
DDR has better latency hence the preference in CPU workloads where lower latency is more important due to variable workloads.
Very true, though traditionally it did refer to GDDR.
And a 5090 has 1.79 TB/s.
So yeah on its own not very impressive.
The large amount of memory means you can serve larger models, or less quantized, but that also means they're going to be slower due to the low bandwidth.
https://www.techpowerup.com/gpu-specs/geforce-rtx-5090.c4216
I'm pretty sure the D270 is meant to be a competitor to the RTX Pro 6000, which also has 1.79TB/s memory bandwidth like the RTX 5090. And it will potentially be a competitor (depending on what the MSRP is) to Nvidia GB10 based systems for 'small' scale AI development. It's not meant to run frontier-size models at the speed you'd expect from something like a cluster of H200 cards.
D270: 1.5TB/s memory bandwidth, 480GB memory capacity, 350W power draw
RTX Pro 6000: 1.79TB/s memory bandwidth, 96GB memory capacity, 600W power draw
Slightly less memory bandwidth, but 5x memory capacity and nearly half the power draw.
Sure, but if you use that memory for the weights of a single model, it'll be ~5x slower.
Yeah but I think this accelerator is more meant for a small scale deployment. This could probably serve a medium moe llm model at reasonable tk/s plus you could in addition to the llm also run a couple of other models on it in parallel like an embedding, reranker, STT, TTS, decision, image, video, … if you deploy this for a small business they might make use of all of these models but not all at the same time and not with 100s of users but this card could bring a serious broad range of AI capabilities for them.
Its bandwidth is around that of 32 DDR5 DIMMS (50ish GB/s). These boards should have built in NICs but they don't because Intel also wants to sell more servers.