Comment by skohan
4 hours ago
Idk about Qwen 2.5 Coder that this is based on, but modern models don't struggle with arithmetic. I see Qwen 3.8 do long-hand math in thinking traces (ie. it outputs 17+35+12=64 in the thinking trace) so I would imagine this is something they train into the model specifically. Sometimes it creates and runs a simple python script to check its work.
It's still strange to me that an ability to perform arithmetic (and not just memorize a giant table of results) could emerge from training on language.
In principle a large enough neural network can provably approximate any continuous function. Since arithmetic can be represented with a concrete grammar, it's not surprising to me that an LLM could contain an abstract representation if trained correctly.