Comment by dexterlagan
2 days ago
Tested the model briefly with my usual eval: a couple of questions on general knowledge most small models often get wrong, then write a fully-featured todo list app in JS, then rewrite the same app in Rust with Tauri. Granted, most models are well trained on basic todo apps, but it gives me an idea of the basic SWE capabilities I can build on. As far as I'm concerned, if it can successfully setup a local git repo, write a todo list app skeleton that works, I can work with it.
SWE: model is strong for its size. It one-shotted the Web app, had no bug. The Rust rewrite only had one bug (reordering didn't work immediately - fixed in one prompt). Committed locally then pushed to my GitHub (https://github.com/DexterLagan/RusTODO). Can't complain. If it can do that reliably, I can use it to make whatever I need.
General knowledge: I usually ask 2 questions many small models get wrong: summarize Operation Trojan Horse by John Keel, give publication year. Summarize the Ariel school incident of 1994. Qwen3.8 got the first question right, along correct publication year, gave glorious detail on Keel's theory, but got the second wrong. It thought that school was located in the USA. Ah well.
Performance on an admittedly overpowered laptop: 15 tokens/s in power save mode on this MacBook M5 Max 48GB, and 30 in performance mode. Perfectly usable for local coding through OpenCode.
Verdict: very nice local and free backup to my usual GPT/Claude/DeepSeek for code. Good for Web searches via Brave search tool calls. What more do you want from a small local model?
> General knowledge: I usually ask 2 questions many small models get wrong: summarize Operation Trojan Horse by John Keel, give publication year. Summarize the Ariel school incident of 1994. Qwen3.8 got the first question right, along correct publication year, gave glorious detail on Keel's theory, but got the second wrong. It thought that school was located in the USA. Ah well.
I'm not sure that this means anything. You're asking a ~27GB file to have losslessly compressed the entire training set (which apparently is a large chunk of the entire internet). That's not possible. Whether it happened to encode these particularly obscure facts losslessly or vaguely isn't really telling you anything about how good a model it is.
Did the model refuse to answer? Did it say that it doesn't know? If not, then it's a fair game in my opinion.
Like most models it makes up things when it doesn't know. I had high hopes for Gemma4, which was said to having 'solved' this particular problem - it didn't. Gemma4 made less things up, did say it didn't know more, but it's still far from perfect. By comparison Qwen3.8 knows more, but still makes things up when it doesn't know. Coding abilities are very impressive however, and yeah these SVG tests do seem to 'scale' or generalize over its general reasoning+coding abilities - at least in JS and Rust. My next test will be to ask it to write some macros in Racket, just to see if it can balance parens. Most, if not all models cannot, no matter their size.
Models don’t know that they don’t know.
4 replies →
[dead]
I actually disagree that this doesn’t mean anything. I understand the contention that it’s not measuring the quality of the model in general, but I think it is measuring something useful.
A good example of this is planning hardware projects - a larger 200b plus model like DeepSeek V4 flash will recommend parts like motors, real time clocks, voltage regulators etc and it will do so providing exact model names and specifications.
I wouldn’t expect a smaller model to encode all of this information, but it is helpful to understand where that cutoff is because it changes what the model might be useful for. It is a very crude way of measuring because it comes down to the balance of training data at sizes this small - but I do think it conveys something that is helpful in real world tasks.
Agreed. It's just to see how much knowledge the model still has for its size. For comparison, no other model of this size has ever given me the correct answer to the Keel question. Even GPT-OSS-120B got it wrong, but it's expected for its relative old age.
I concur, treating models as question and answer machines and judging them on recall is meaningless, unless you're measuring quantisation impact on a foundation model maybe.
100% agree, it's really just for kicks&giggles. The fact that the model answered correctly, unlike every other model of its size before it, still pleasantly surprised me.
> on an admittedly overpowered laptop: 15 tokens/s in power save mode on this MacBook M5 Max 48GB, and 30 in performance mode.
Okay, so I'm here to brag a little. I love that I also get 30 t/s on $1500 of decade-old hardware: dell r720 w 2x tesla v100s!
Which setup did you use? MLX/GGUF, Quant, Engine (e.g. llama.cpp or MTPLX, etc)? There’s so much variety these days.
It was in LMStudio (llama.cpp), Q4 by Unsloth. Applied the recommended defaults published by Unsloth.
Meaningless yet fun fact: DeepSeek V4 Pro 0813 made a much worse icon for the same app, and only produced an SVG I had to convert manually to .png. Qwen3.8 made a perfect icon in .png. I don't yet know how it did it, but it did it.
Qwen3.8 also seems to know French quite a bit better than Copilot, at least on common expressions. I have yet to run more tests for languages, but I'm baffled by its finer accuracy on the few sentences I asked it to translate, which were botched by Copilot. Go figure.
copilot isn't a model. it's a model hoster.
> write a fully-featured todo list app in JS
Would you mind sharing how you prompt this? I'm not a developer myself (just someone who occasionally dabbles, though most of my coding was pre-LLMs) and curious to see how much info/instruction you consider necessary to test them making an actual app (albeit a simple one).