Comment by mlmonkey
18 hours ago
In the clip with a marble bust dropped on a watermelon, at this point https://imgur.com/a/IHLnkUs the bust should've toppled over, if the physics were properly modeled.
18 hours ago
In the clip with a marble bust dropped on a watermelon, at this point https://imgur.com/a/IHLnkUs the bust should've toppled over, if the physics were properly modeled.
This is a reflection of a complaint I frequently make about so called "world models". It's more accurate to call them video game models. They don't simulate the world, they simulate video games. For that reason a common issue with them is that the physics are video game physics rather than real world physics.
I'm sure you're probably right about video games being used in training and, as a result, the models reflecting video game physics: No air resistance, no static/dynamic friction curve, etc.
However, I don't think that's the problem in the example of the marble bust not toppling. It would have toppled in a video game engine as well; they model gravity and inertia just fine.
I think what's happening is the model isn't being a world model at all, it's being a language model. It's trying to make the "scripted" thing happen, and it probably "feels" like the script should end in the bust on the pedestal.
Yes this is right. Something about the way it works under the hood means it wants: 1. bust falls 2. it crushes the watermelon 3. it teeters 4. it settles on the plinth. It is sort of interpolating between these states, which it feels are a realistic sequence of events, not actually attempting to model what would happen in a real "world".
It also failed to properly model the physics of knights riding wyverns into battle.
Which is fine for a video game. Not at all fine if you want to train robots with it. Trying to combine both use cases into a single model is doomed to failure IMO.
I would not be so opinionated, modern coding agents derive from chatbot technology. Very different problem domain. They share language but language is universal so that doesn't say much. Then they figured out that the pretraining is a good starting point for RLVR. Who's to say a more cinematic imagination can't be a good starting point for a world model? It doesn't need to be the whole solution.
If they don’t get ahead of this our robots are going to be sitting ducks against knights riding wyverns smh
People are arguing if the marble bust should've toppled over or not, completely missing the fact that marble is brittle and the bust would have surely cracked
How often do things in the real world not behave exactly as you expected them to?
I feeling I get from this is that this could be the mental model for a robot in the real world, continuously corrected by actual events.
We all have a world model in our heads of what we think is going to happen, we adjust quickly (or not) based on what does. This is pretty incredible stuff.
Busts rarely hang in midair at a 45° angle.
As an aside, why fall into the role of being an apologist for google-hype and Big Tech?
You're missing the forest for the trees. The less than amazing physics isn't the point.
When you have a conversation, you have an idea in your mind of how the conversation is going to go. You predict what the other person is going to say before they say it.
When you are driving, you predict what the other drivers are going to do, you predict what signs say, how long you have until the green. If you see two red lights attached to the back of a car flash on, you may predict that the car is about to slow down and you start slowing down before their speed has changed.
Forget video games and movies. I believe these models will be that inner predictive model for robots.
> How often do things in the real world not behave exactly as you expected them to?
They wouldn't be called "Laws of Physics" if obeying them was optional. If your center of gravity is outside your base, you _will_ topple over. That's just how gravity works.
Your expectation is the point, not the physics. I'm not arguing that their model has an amazing grasp of physics. I'm arguing that this can be used as a predictive internal model for a robot.
I agree but I feel like the exaggerated physics were intentional and fitting with the video theme. If they had wanted it to topple, they wouldn't have posted the video.
They were demonstrating steering.
So what, the world model failed to model physics. The thing they said it did elsewhere. It's a valid critique
That's the thing about machine learning over time, and the big hole in the whole world model idea. Time and time again we make simulations and train models and we think that loop will work. It never does.
And why not? Because the first model that can actually learn to handle the simulation not only can do the simulation internally too, but it can do it better than out simulation.
It's where (ex-)"AI luminaries" go to die. It's a way to make glacial, slow progress in the correct direction while never actually getting anywhere.
So we get to a point and people keep making complicated simulations of one kind or another ... and then someone (some model) who's never used the simulation, and doesn't need it for model training gets by far the highest score on every benchmark and everyone agrees the simulation was kind of beside the point.
It depends if the goal is a more accurate simulation, or if the goal is a tool for creatives to build games or content.
If it's the latter, I don't see a 'hole'.
My point was that there is also the possibility of AI intelligence improving to the point that there isn't a difference anymore. That whatever beats the next set of benchmarks (which of course won't be the simulator or world model) is so smart it is both a better solution to the problem the world simulator poses and a better world simulator all in one and a better tool for creatives to build games or content.
This has already happened to other ML fields. For example, there's not much point anymore to ASR models (just feed audio into LLMs) and the space that TTS models occupy is fast disappearing. Oh and it's not just that Qwen 3.6 is a good ASR in addition to being good at coding mario. Doing things this way, not using an ASR model is fundamentally more expressive. A whole bunch of companies started up on ASR + TTS "towers" (lots of models, Pipecat style) as they're called, and they're dead companies walking. Their tech has been surpassed by just bog-standard generic LLMs in a way that they will never get the upper hand again.
A chain of models approach, a separate ASR, will never do this: "you said you agree but I can hear you're sarcastic, so I'll explain again". Also getting a conversational tower to switch languages when the user does that (or when it makes sense in a language course) is not quite impossible but very, very hard. And yet feeding audio directly into Qwen will almost magically solve these problems better than I will ever do it (lightly finetuned). It's fundamentally better in every way (except perhaps speed, of course. Although arguably such a tower needs an LLM too, it'll just have way less tokens)
This point has long been passed for image recognition. Feeding a photo or screenshot into a full LLM along with a request on what to do beats the crap out of any OCR, image recognition, ... model every time these days.
And now there's also hints of world models just being bypassed. Just giving a finetuned model the last 50 video frames (surprisingly, that fits in context, especially at reduced resolution), movement statistics, motor power, status, ... of a robot and asking what to do actually works (with heavy finetuning, but I think it'll start working without). Well ... AND then distilling that knowledge into a much smaller model because B200 racks are more than a bit to heavy for a logistics robot to carry around. But we'll get over that.
2 replies →
yes. I feel Google's video physics has always been bad. slow, aritifical, unnatural.