AMD 7900 XTX with Vulkan here as well, wasn't faster on my test either. Might be much different on Nvidia though.
I assume the limit for me is memory bandwith, as the 7900 XTX has the same bandwith as the 3090 from what I can gather and I already reached ~60 t/s with Unsloth. Those would fit with the numbers Byteshape has for their cards.
4090 and 5090 have much higher bandwith apparently, so on those cards you can probably get much more out of the kinds of performance improvements they are doing.
AMD 7900 XTX with Vulkan here as well, wasn't faster on my test either. Might be much different on Nvidia though.
I assume the limit for me is memory bandwith, as the 7900 XTX has the same bandwith as the 3090 from what I can gather and I already reached ~60 t/s with Unsloth. Those would fit with the numbers Byteshape has for their cards.
4090 and 5090 have much higher bandwith apparently, so on those cards you can probably get much more out of the kinds of performance improvements they are doing.
Surprised its not meaningfully slower.
Vulkan and ROCm paths are missing a few optimized versions of the quants they're using.