Comment by swyx
1 day ago
this is the right answer. ultimately text and general llm usage incl coding is far more internally complex than just memorizing the looks and composition of concepts placed in an image. dont forget that most of our reasoning benchmarks basically require the llm to have a search engine and calculator in its own weights.
These diffusion models are not just memorizing the looks and composition of concepts. These models are increasingly capable of representing advanced world knowledge. Things like shadows require a comprehensive, if indirect, understanding of physical light modeling. Things like weight/size estimation, etc., which are all important to a cohesive generated image of natural phenomenon, require an adequate understanding of physics.
Some of this understanding is in the input embeddings themselves, embedding models do a lot of heavy lifting behind image generation models.
I posit that you can theoretically achieve the same capability in a model trained on the right sequence of "images" as you can purely text; both are turned into embeddings and can ultimately convey the same exact relationships. I don't think it's very useful to try and create delineations between text and image inference, especially as more models move towards native multimodality.
i think well argued and a reasonable hypothesis (esp considering deepseekocr and most omnimodel research), but you are wrong heheh. approximating instantaneous physics is, while nontrivial, still computationally light vs the stuff we ask LLMs to do.
look at any study of llm depth vs multihop reasoning capability (here, just to help https://www.latent.space/p/ainews-death-of-params-zai-ceo-ji... but theres a long line of others).
i will grant that people do do "reasoning in image models" type benchmarks, and it's not nothing, but its kid stuff vs what we make LLMs do, and the only way you get image models that basically do reasoning is you glom a diffusion head on an LLM backbone, which is basically gpt image 2.0 and above
none of which is to dispute the original question which is perhaps "why are image models so smol vs LLMs" - which i agree with and wish there were funding for scaling them - but the abject failure of sora indicates that perhaps our data isn't nearly good enough (again pointing to another thing - much easier to pretrain and posttrain LLMs given quantity of text data)
What specifically are you claiming is wrong about my comment? I never made any comparison to LLMs, I spoke about training multimodal models and how image models are doing not just superficially memorizing the looks and composition of concepts.