Comment by dimatura

I don't think whether LLMs use only the last token, or all past tokens, affects LeCun's argument. LLMs already used large context windows when LeCun made this argument. On the other hand, allowing backtracking does. Which is not something the standard LLM did back when LeCun made his argument.