Comment by julius
19 hours ago
I always heard of GB/s as most important number ... AI told me their 8800 MT/s on 8 Byte, 12 DDR5-Channels means 845 GB/s. A Nvidia RTX 4090 has 1008 GB/s. Nvidia B200 has 8000 GB/s. Is this the right way of looking at it?
You have to load the model weights into VRAM over PCI-E (from RAM). So the (PCI-E) bandwidth strongly affects time to first token.
You have to run inference on the GPU by reading and writing to VRAM. So TFLOPS of the compute matters, and bandwidth to the VRAM (Always integrated with the GPU, rarely a bottleneck), and this strongly affects tokens/s
If you're doing training workloads or offloading to system RAM, it gets more complicated. (And mostly bound up trying to feed compute on time)
(Edits for clarity.)
> So the (PCI-E) bandwidth strongly affects time to first token
On dedicated inference hardware I'd expect model weights to never leave the RAM, and you'd probably load them on startup before even starting to serve requests