Comment by user43928
5 hours ago
If we trained a LLM on such texts that only use "none, one, few, some, many" in their language, wouldn't it likely learn representations of individual quantities and arithmetics anyway?
Provided the training data was extensive enough and training rewarded solving problems that require mathematics.
Interesting question. I don't think so. In spite of what they're achieving right now, LLMs remain token prediction algorithms. We're speaking of going from a world where the concept of 'multiply' simply didn't exist to it being invented and formulated.
I also don't think the people behind the LLM companies think this is the case either. If it were then it'd make so much more sense to drop the current regime and instead move to the most basic systems trained on nothing but the most fundamental first principles and have them try to derive everything from there. It'd ostensibly lead to far more reliable systems with little to nothing in the way of bias. It'd also likely be vastly cheaper than the current practice of trying to train on essentially all consumable knowledge.