Comment by unddoch

It needs to gather a bunch of inputs and feed into the remote GPUs correctly and as efficiently as possible. Lots of room to optimize under the constraints that users love cheap tokens, demand a fair billing structure, and want responsiveness and low latency.