Comment by Grombobulous
10 hours ago
Maybe this is a dumb analogy but is that kind of like saying “well EVs will still use lubricant! Cars that use oil are here to stay!”?
The point being that the way we heavily rely on LLMs specifically right now is likely to cease to be the case, and that in the future they’ll be almost like an irrelevant component.
There's far more shared concepts between them than you think.
You should check the code of both world models (JEPA class for example) and compare to GPT. Many of the tricks stay the same, representation is still embeddings, there is a loss function, etc.
The exact architecture will change, but unless there's a new discovery in that area, we've cracked the text component already. We're hitting the limits of LLMs because of the intrinsic limits of text as a medium. But the way we work with text is pretty much settled, fundamentally.
This is very interesting to me, and I have new things to read into, thank you.