Comment by r0b05
13 hours ago
It would be more interesting to know how many simultaneous users this setup can serve. Otherwise I can just say it serves 500 users but not all of them use it at the same time which doesn't communicate the right level of detail.
Depends on TTFT and tokens per second you want.