Comment by inkcapmushroom
13 hours ago
For several of these examples the details change if something goes out of frame and returns. A second castle is added in the dragon one. The people walking away from the camera change in the train platform one. The Hong Kong moped rider doesn't have a helmet and then he has one.
It's very good at maintaining consistency for things that stay in frame, but I don't know that this points to it developing a strong "world simulation" like they claim.
It's like its simulation model is confined to the renderable area. At the cost of increased computation, it would not be hard to generalize the technique to rendering a sphere centered around the POV to keep some of those artifacts at bay; however, the fundamental limitation of "object permanence" remains if objects can evolve chaotically as soon as they are blocked from view by another object. For that to be made stable, you really need a whole 3D model of the scene, and the cost increase there might be huge.
> While it can maintain coherent environments over short sessions, visual details and object permanence can still shift or drift over time—especially as a simulation runs longer, after multiple complex prompt edits, or when navigating throughout the scene. Complex physical interactions, real-time audio accuracy, rendering legible text, spatial accuracy, and long-term consistency are active areas of ongoing research.
If it was actually not hard, they would have made their demo with it. I don't know the technical details, but I expect a product reveal like this to show the thing at its absolute best.
It is possible that it is not hard, but they didn't have time to implement it for this release. It happens all the time in fast-moving industries.
[dead]
I am not sure you need something so complex. If objects on scene could be tagged with their location, and the camera movement tracked to identify when it brings the same objects back into scene, then the model can use its positional information to validate if the next rendered scene contains all the objects that should be expected.
Though, I am not sure well this retains temporal stability. I am just thinking about how in 3D rendering, objects are tracked as individual things and only culled, but all the data about them still exists.
Maybe the models need to reason as a 3D space first, then turn chosen objects into renderings?
The trick is to render a larger frame in lower fidelity and then only render a smaller output frame in full fidelity.
Or render a low fidelity 3d model for coherence and then raytrace that onto a 2d frame that is then rendered at higher fidelity.
1 reply →
> For that to be made stable, you really need a whole 3D model of the scene, and the cost increase there might be huge.
I suspect it is, because the 2D -> 3D models I've seen are terrible, but more importantly the 2D -> 3d models are terrible. Noteworthy because I suspect the number of people who have 3D scans laying around is tiny.
What you're describing is vaguely how video games work, to my understanding. There may be interesting parallels if that becomes a technique (like how distant objects are sometimes still rendered, but at a lower resolution/polygon count).
> What you're describing is vaguely how video games work, to my understanding.
Your understanding is broadly correct. There is a 3D world, and the GPU computes how it looks like from a particular viewpoint.
In essence, objects are painted back to front, so that in the final image every pixel shows the nearest visible object. The details get complicated, but the idea is simple.
> fundamental limitation of "object permanence" remains if objects can evolve chaotically as soon as they are blocked from view by another object.
they got this working decently well for video over a year ago, maybe closer to two years ago at this point, time goes fast. so i assume it'll translate to progress in Genie too
This sounds like bunch of gibberish you made up
[flagged]
I don’t understand why these world models aren’t designed to be backed by an updatable world state. It could probably just lay out the world onetime and reference it as needed vs trying to invent from wholecloth via diffusion. Object persistence!
> It's very good at maintaining consistency for things that stay in frame
I noticed several inconsistencies in things that stayed in frame. See the wall on the bridge in the Van Gogh example as they walk past it to see the other side.
I bet it's configurable and just a matter of how much resource you want to allocate. If it's truly a world model then it seems to me that it's precise benefit is that it can keep track and order in things and they just choose to accept some noise/chaos/randomness/? for the demos when things move out frame? Does that sound sensible? I'm just guessing but don't feel it's a hard limit.
And these are the examples they chose. I have to wonder if they add some with mistakes on purpose.
If it's anything like their other anything related to AI, we're seeing cherry picked "best possible" examples.
It just doesn't have object permanence yet!
[flagged]
[dead]