Comment by mirekrusin

4 hours ago

27b runs perfectly fine on 2x 24GB at ~100 t/s (4090) with speculative decoding on 8 bit quants

1 comment