Comment by RandomBK

I'm curious to hear what bottlenecks you encountered in the traditional path. Of all the compute and data shuffling involved in LLM inference, I would have thought shuffling the raw input/output around would have been a trivial part of the overall cost, and thus not a big optimization target?