Comment by dist-epoch

Jev claims 70-500 ms latency, including for the first request. This requires some clever engineering at least, which will take a little to duplicate.

Maybe first request is unbatched, to have fast prefill, and the subsequent ones are batched.

They also don't restrict your prompt. You can have a dumb one, where you put the variable data at the front, and the details on how to process it at the back, thus you bust the user-part of the KV cache every request.