Comment by torginus

What you said doesn't contradict me, and doesn't refute my point.

For images and audio, you still need to predetermine an encoding, then pretrain to learn an embedding. This embedding will try to replicate the input distribution - so if you trained it on Google Street View and scanned documents, its representation will be grounded in only those.

I would even claim that this approach is somewhat counterproductive, as images are far more information dense, containing tons of concepts

While LLMs do have some ability to learn to use their embedding space in non-predetermined ways, they still lack the ability to pick an efficient embedding.

So I guess, a nice thing is that interpretability is baked into this approach to some degree, and humanity has proven through its existence, that you can do a lot with just text, but this approach is still predetermined.

I guess this is what LeCun's JEPA is about, that the AI gets to learn the representation on its own as well.