Comment by vidarh
4 years ago
I wonder how these language models would do if we tried to teach them maths the way schools do: Feed them explanations first, then endless sequences of toy problems, see which they got wrong and feed them corrected examples back in.
I'm not at all surprised they don't do well at maths, because while there are maths texts online, I doubt there is enough material to give these models the same experience of repetition and reinforcement to help sufficiently generalise an understanding of the underlying rules.
Generating solved math problems is trivial, like making AlphaZero play itself in chess. Sparse Data is not the problem. Refusing to use it is.
I don't think it's so much a refusal, as that it's not been a sufficient priority for anyone before. As the article points out there are now a few training sets which includes math problems, and models which do well on them. But the remaining problems seems to be with basics which humans tends to learn to do consistently with a lot of repetition, and it'd be interesting to see those datasets extended to the very simple.