Comment by londons_explore

It is the tail latency * number of layers.

If we assume a network stack can be tuned to a tail latency of 10 us, that caps the tokens per second to ~2000 (assuming no predictive stuff).

Still seems possible. Note tail latency doesn't really affect throughput, and therefore doesn't matter for $/GPU/hour.