Comment by hashmap
10 hours ago
You're missing the point here. He's not talking about whether or not they can learn facts or inferences derived from the text itself, but the more holistic intuition that results from learning from something like an embodied experience in the physical world. GPT-6 Astras web demo homepage thing is an example. It chose euclidean rather than quaternion for letting a user rotate the galaxy thing, and anyone who has ever used hands to rotate something would immediately recognize on trying it that something is fucked and you shouldnt do that. Thats the kind of common sense physics that is inherently beyond these llms and I run into it ALL the time in vr programming.
To be fair, LLMs can still derive those kinds of things from text, at the very least from your own comment if it made it to the training set though I'm sure it is mentioned in a lot of other places already. Many of this type of mistakes went away after reasoning was introduced.
But I'm sure you can still find tasks that they will have difficulty solving, involving the most fundamental concepts that can only be experienced in the physical world to be understood well, like left and right, near and far, hot and cold, heavy and light, etc.
Yup it lacks common sense because it doesn’t ‘understand’ reality - how could it? It doesn’t touch it like we do everyday. It has access to what is a model of reality via data.
The good designer understands culture, tastes and preferences as they evolve in real time. That’s why llm as design tools haven’t displaced the good designers.