Comment by jedbrooke

Running at about 7-8 tok/s (~60 tok/s prefill) on a Mac Mini M2 16GB.

So far feels smarter than Bonsai 1 27B, it’s slightly larger than the Q1_0 quant. Super exciting stuff :)