Comment by Yajirobe
2 days ago
Did the model refuse to answer? Did it say that it doesn't know? If not, then it's a fair game in my opinion.
2 days ago
Did the model refuse to answer? Did it say that it doesn't know? If not, then it's a fair game in my opinion.
Like most models it makes up things when it doesn't know. I had high hopes for Gemma4, which was said to having 'solved' this particular problem - it didn't. Gemma4 made less things up, did say it didn't know more, but it's still far from perfect. By comparison Qwen3.8 knows more, but still makes things up when it doesn't know. Coding abilities are very impressive however, and yeah these SVG tests do seem to 'scale' or generalize over its general reasoning+coding abilities - at least in JS and Rust. My next test will be to ask it to write some macros in Racket, just to see if it can balance parens. Most, if not all models cannot, no matter their size.
Models don’t know that they don’t know.
> know that they don’t know
And we are waiting for architectures that do - because it's duly.
Honestly it seems like a job for the harness, rather than the model. Sample the model with the same question, perhaps with varying temperature (?), and use that to establish a degree of confidence in the answer. If the model provides very different answers every time, respond that it doesn't know. If it responds with the same answer usually but a different answer sometimes, respond with moderate confidence. If the model always responds with the same answer, respond with certainty.
2 replies →
[dead]