Comment by pllbnk
8 hours ago
Benchmarking proprietary models is useful but it leaves a lot unsaid because a lot of information is hidden. We have seen how 27B local model (Qwen 3.8) can stand its ground against these flagships in many cases. For all we know Fable 5 could be a masked Opus 4.6 with a few knobs tuned and thinking turned up to the max++. Harnesses keeping these models in check is the key to solving problems, not the models themselves, in most cases.
Qwen 3.8 is both impressive in what a 27B model can do and then hilariously stupid the next second, literally smashing into the wall repeatedly when the door is 3ft to the right.
That said it's the first local modal useful enough to keep around and since I have zero interest in remote/hosted models actually the first time I've used one to do useful things (rather than just poking them with a stick to get a feel for what they can and can't do).
Yeah, this lines up with my experience. It will overkill things and then dig a hole for itself and fall in.
But did you test Muse Glimmer? IMO it is really worth comparing the two, at least. I really find it interesting (and while it is slower at outputting tokens, it solves problems better and usually even faster than the 3.6 35B MoE).
So far it happens to be the only one I've put code into production from (though I have done loads of useful self-teaching research with the Qwen models and I am grateful for them)
I did, Muse came close but what impressed me about Qwen is it’ll push back if it thinks it’s right even when it isn’t, I can work with that, Muse tended to flip between states too easily/too much.
It is a good model but Qwen (at least for the things I use it for) edges just ahead, it seems much better at the “rip this apart, suggest improvements, touch nothing” use case where I can use it as a second set of eyes, I don’t agree with all its suggestions but it catches enough to be worth running while I grab coffee, it also seems to follow instructions better in terms of outputting more what I asked for than what it thinks I asked for.
Qwen is the only local model that said in its thinking “I think the user is pushing me to see if I’ll suggest something even though I have nothing to suggest, I should just say that” and then did, caught me off guard, they didn’t do that so readily 6mths ago.
The ISTA version is also comfortably able to fit on a 7900XTX with a good amount of space left for context and is decently fast given the AMD cards are not as fast as nvidia cards of same era/rough price, didn’t buy it for AI but it’s surprisingly capable mostly because 24GB at 960GB/s is still a lot of bandwidth compared to everything but nvidia cards.
1 reply →