Comment by eveningtree

The point of speed is to increase throughput. What the point of all this speed, if overall throughput is still so low?

This doesn't work for my use case at all (code generation).

These bursts of speed might work well for workflows that need bursts of quick decisions, followed by silence. But these workflows have needed provable determinism to som extent, so I haven't been using llms for those use cases. And I don't see myself using llms for them in the future too.