Comment by CamperBob2

One problem for your argument is that transformer networks are not, and weren't meant to be, calculators. Their raw numerical calculating abilities are shaky when you don't let them use external tools, but they are also entirely emergent. It turns out that language doesn't just describe logic, it encodes it. Nobody expected that.

To see another problem with your argument, find someone with weak reasoning abilities who is willing to be a test subject. Give them a calculator -- hell, give them a copy of Mathematica -- and send them to IMO, and see how that works out for them.