Comment by xg15
3 days ago
There are other next token prediction algorithms such as markov chains or HMMs that also "fit the same interface", but are vastly simpler than what LLMs use.
I've seen various posters talk about "stochastic parrots" or about how LLMs were just "using very simple word statistics" to get their results, which sounded to me very much as if they thought of LLMs simply as glorified markov chains. I would consider that a big misunderstanding.
No misunderstanding at all.
"In machine learning, the term stochastic parrot is a metaphor that frames large language models as systems that statistically mimic text without real understanding." https://en.wikipedia.org/wiki/Stochastic_parrot
"Large Language Models as Markov Chains" https://arxiv.org/abs/2410.02724