Comment by voiceeh
14 hours ago
Not if your use case needs speed. For one of my products I can't use an LLM that has a p99 of >700ms for TTFT.
14 hours ago
Not if your use case needs speed. For one of my products I can't use an LLM that has a p99 of >700ms for TTFT.
If it could output 1k tokens per second but needed 4 seconds to produce the first batch of 4k, would that not be viable?