Comment by KaiserPro

Thats where the research is leading.

The issue is context. trying to make an AI assistant with just text only inputs is doeable but limiting. You need to know the _context_ of all the data, and without visual input most of it is useful.

For example "Where is the other half of this" is almost impossible to solve unless you have an idea of what "this" is.

but to do that you need to have cameras, to use cameras you need to have position, object, and people tracking. And that is a hard problem thats not solved.

the hypothesis is that "world models" solve that with an implicit understanding of the worl and the objects in context