Comment by Aurornis
5 hours ago
The Bonsai models are really bad when you actually use them for more than short responses.
Their marketing made it look like a breakthrough, but in my experience it’s just the next step down from the Q2 quants in both size and quality.
Q2 quants are already not very useful in my experience. The Bonsai models are even worse.
If you only need 80% plausible outputs that don’t need to reference a lot of context they can be useful. If you try to use them for real tasks it feels like time warping back to 2023 when you LLMs were barely useful if you babysat every word of the output.
Have you actually used Bonsai 2 though and not just the original Bonsai? The experience is vastly improved but still requires a custom llama.cpp fork to use as of right now.