Comment by famouswaffles

>It's a well known problem that it cannot consistently add arbitrary numbers.

Neither can humans. Calculators aren't just for saving time. Humans regularly mess up arithmetic.

Can you add arbitrary numbers any better than GPT-4?. Test it out yourself.

>As for feedback: This is obviously not part of the training process so can't be part of what is "learned".

A token that is part of the process of an ongoing recursive computation in text being fed to a transformer is definitely part of training. Training directly incentivizes being recursive in token space.