Comment by bjourne
3 months ago
Well, that an autoregressive model has parameters does not mean it is not autoregressive. LLMs are not Markovian.
3 months ago
Well, that an autoregressive model has parameters does not mean it is not autoregressive. LLMs are not Markovian.
At no point have I argued that LLMs aren’t autoregressive, I am merely talking about LLMs ability to reason across time steps, so it seems we are talking past each other which won’t lead anywhere.
And yes, LLM can be studied under the lens of Markov processes: https://arxiv.org/pdf/2410.02724
Have a good day