Comment by n_u
6 hours ago
> The total number of concurrent requests represents the number of simultaneous requests moving through the pipeline at any point in time. Unlike a GPU batch, those requests don’t wait for one another to finish, they stream through the pipeline continuously.
What do they mean by “unlike a GPU batch”? It’s not a hardware limitation of GPUs right? It’s just the scaffolding to do this in software isn’t available right
It's describing a different type of parallelism - pipeline parallelism. The network is laid out spatially on the wafer in CB's case. So weights are stationary and activations flow. A GPU is going to process the network one layer at a time. GPUs also use other types of parallelism (tensor and expert mainly in inference). Pipeline parallelism doesn't make much sense on a GPU for inference.
I think the point is they need several wafers to fit the model. Data parallelism between requires too much bandwidth (they only have so much edge real estate for io) so you have to do pipeline parallelism with say one layer per wafer. No one does it on gpu unless you can’t fit the model in one fast network domain (which is 72 GPUs nowadays for Nvidia). There is a Dwarkesh episode from like two months ago that explains this reasonably accurately.
From reading xpct's comment in this thread it seems like they use effectively a very deep and wide pipeline by tiling the cores without central clocking or dispatch. In theory if a GPU is centrally organized it could simulate this behaviour, but I imagine that creates transfer overhead. Just speculating though.