Comment by esafak
8 months ago
The models can't tell when they shouldn't extrapolate and simply need more information. Which rules can be generalized and which ones can't. Why shouldn't a method `doWhizBang()` exist if there methods for all sorts of other things?
When I was young, I once beamed that my mother was a good cooker. It made perfect sense based on other verbs, but I did not know that that word was already claimed by machines, and humans were assigned the word cooks. Decades later, I had the pleasure of hearing my child call me a good cooker...
This made me think – the fact that the underlying rule that nouns e.g. representing activities such as cooking can be formed from the corresponding verb via the suffix -er breaks in this case is just a historical / cultural artifact of languages, a type of completely unnecessary complication from the machines' standpoint. Maybe LLM hallucination might also partially be caused by this exception-based social modelling forced via our training data into all model architectures?