Comment by dexterlagan

Like most models it makes up things when it doesn't know. I had high hopes for Gemma4, which was said to having 'solved' this particular problem - it didn't. Gemma4 made less things up, did say it didn't know more, but it's still far from perfect. By comparison Qwen3.8 knows more, but still makes things up when it doesn't know. Coding abilities are very impressive however, and yeah these SVG tests do seem to 'scale' or generalize over its general reasoning+coding abilities - at least in JS and Rust. My next test will be to ask it to write some macros in Racket, just to see if it can balance parens. Most, if not all models cannot, no matter their size.