Comment by _aavaa_ 15 hours ago It’s cheaper sure, but it’s very slow. It’s not a drop in replacement 2 comments _aavaa_ Reply natrys 15 hours ago I think we don't have a good draft model for better speculative decoding yet (e.g. DFlash 2). Once we do, it will be faster. _aavaa_ 14 hours ago It very well could be faster, but right now it isn’t.
natrys 15 hours ago I think we don't have a good draft model for better speculative decoding yet (e.g. DFlash 2). Once we do, it will be faster. _aavaa_ 14 hours ago It very well could be faster, but right now it isn’t.
I think we don't have a good draft model for better speculative decoding yet (e.g. DFlash 2). Once we do, it will be faster.
It very well could be faster, but right now it isn’t.