Comment by aabdi

I don’t think it would be surprising that people want to write their own kernels.

A big problem with the existing engines like llama or sd is that they don’t support optimal graph compilation. Usually this means about a real 2 or 3x multiplier loss relative to optimal. Cuda graphs do okay but they still leave a lot on the floor

It’s usually worth it to optimize in that context if you are willing to peer into the mechanics.

Of course that’s expensive. You need to know how to appropriately pipeline and merge your kernels.