Comment by tomrod
11 hours ago
I wonder how quickly the latency between machines would add up. I'll admit this kind of process routing is interesting from an OR perspective.
11 hours ago
I wonder how quickly the latency between machines would add up. I'll admit this kind of process routing is interesting from an OR perspective.
It is the tail latency * number of layers.
If we assume a network stack can be tuned to a tail latency of 10 us, that caps the tokens per second to ~2000 (assuming no predictive stuff).
Still seems possible. Note tail latency doesn't really affect throughput, and therefore doesn't matter for $/GPU/hour.