Comment by gdiamos

vLLM is originally marketed as paged attention, but in hindsight, separating the web server and GPU process, continuous batching, kv caching / chunking, and a huge model library including low precision mattered more.

I wonder how much it would cost to vibe code the whole thing from scatch?

I wonder how much better models need to get before such a thing wouldn't look like code vomit?