Comment by saithound
7 days ago
No, context windows are not arbitrarily long and complex. The set of possible context windows is a large finite set. The mathematical theory of Markov chains does not depend at all on what the elements of the state space set look like. The same math applies.
You argue LLMs are Markov chains because the context window is a 'large finite set.' But the physical configuration of the human brain is also a large finite set. We have a finite number of neurons and synaptic states; we do not possess infinite memory or infinite context.
Therefore, by your strict mathematical definition, a human is also a discrete-time Markov chain.
And that is exactly my point: If your definition is broad enough to group N-gram lookup tables, LLMs, and Human Beings into the same category, it is a useless category for this discussion. We are trying to distinguish between simple statistical generators and neural models. Pointing out that they both satisfy the Markov property is technically true, but structurally reductive to the point of absurdity.