Comment by latentsea

8 hours ago

You can run a 4 bit quant with this. Personally, I switched to running IQ3_XXS and am getting better outputs than 27B and at faster speeds.

0 comments