Comment by SwellJoe
11 hours ago
Bonsai 2-bit quantization is notably dumber than Qwen 27B at eight bits or even four bits. I doubt you'll get this level of quality even with five of them.
11 hours ago
Bonsai 2-bit quantization is notably dumber than Qwen 27B at eight bits or even four bits. I doubt you'll get this level of quality even with five of them.
Dumber yes, but not by all that much. If the multiple agent thing really improves the quality by as much as is described here, the quality loss should be nearly imperceptible.
I’ve tried the Bonsai quants specifically because I want to run a lot of parallel requests.
The swarm strategy can’t overcome the dumbed down model problem. You just end up with your swarm of agents all going in the same dumbed down loops.
Throwing Ralphs at the wall in parallel
1 reply →
I think that requires testing, rather than assuming. There's only so far a harness can go.
Not sure if I agree. Remember the error rates stack up every turn of the conversation. If a 2bit model is 10% more wrong than the fp16 version, over 10 turns that adds up