Comment by jszymborski
4 hours ago
No, it's what I get repeatedly. Recall that it's QB_0 though. I'll get back to you when the Q4_K_M quant finishes downloading.
EDIT: I get 259 tok/s with the Q4_K_M quant
4 hours ago
No, it's what I get repeatedly. Recall that it's QB_0 though. I'll get back to you when the Q4_K_M quant finishes downloading.
EDIT: I get 259 tok/s with the Q4_K_M quant
I assume you mean Q8_0, not QB_0? (was confused, as I've not obviously seen a QB_0 before and can't find one now).