Comment by vatsachak

That's true, but he's still correct, it's just that the context is now so large that only people using agent loops see "context rot"

His other criticism of LLMs that I like better is that they try to predict tokens instead of learned embeddings. Tokens are arbitrary and in order to decode LLMs you need technical analysis (see mechanistic interpretability).

With JEPA models so far, it seems that PCA on latent vectors suffices.

tldr: embeddings have a lot more room for improvement