Comment by rao-v
19 hours ago
I don’t understand why these world models aren’t designed to be backed by an updatable world state. It could probably just lay out the world onetime and reference it as needed vs trying to invent from wholecloth via diffusion. Object persistence!
It must be due the training data. YouTube videos don't have the metadata of all the objects and where they are/were.
Kind of makes me think that autonomous vehicles will eventually gather better dataset (although limited to roads!), since they do have 360 degree cameras and lidar, so they can track objects before & after they are shown on certain view.
Couldn't they combine the training data with a ton of synthetic data that allows them to understand how 3D engines work?