Comment by roenxi
1 day ago
To make a rather speculative argument - since an LLM should be able to describer an image losslessly, images are a subset of what an LLM needs to understand. It is possible that the statistical model being learned for images is just simpler.
The get a proper LLM they have to model all of human knowledge regarding patterns, physics, biology, social interaction, skills, history, etc, etc. For images they have to recall what a dog looks like in a still photo. The image models aren't even necessarily good at a wide range of novel situations; usually they get used for stock scenes.
Natural evolution has also had a much easier time coming up with vision models (everything with eyeballs) than complex language models (as far as we know only a smallish family of monkeys).
this is the right answer. ultimately text and general llm usage incl coding is far more internally complex than just memorizing the looks and composition of concepts placed in an image. dont forget that most of our reasoning benchmarks basically require the llm to have a search engine and calculator in its own weights.
These diffusion models are not just memorizing the looks and composition of concepts. These models are increasingly capable of representing advanced world knowledge. Things like shadows require a comprehensive, if indirect, understanding of physical light modeling. Things like weight/size estimation, etc., which are all important to a cohesive generated image of natural phenomenon, require an adequate understanding of physics.
Some of this understanding is in the input embeddings themselves, embedding models do a lot of heavy lifting behind image generation models.
I posit that you can theoretically achieve the same capability in a model trained on the right sequence of "images" as you can purely text; both are turned into embeddings and can ultimately convey the same exact relationships. I don't think it's very useful to try and create delineations between text and image inference, especially as more models move towards native multimodality.
i think well argued and a reasonable hypothesis (esp considering deepseekocr and most omnimodel research), but you are wrong heheh. approximating instantaneous physics is, while nontrivial, still computationally light vs the stuff we ask LLMs to do.
look at any study of llm depth vs multihop reasoning capability (here, just to help https://www.latent.space/p/ainews-death-of-params-zai-ceo-ji... but theres a long line of others).
i will grant that people do do "reasoning in image models" type benchmarks, and it's not nothing, but its kid stuff vs what we make LLMs do, and the only way you get image models that basically do reasoning is you glom a diffusion head on an LLM backbone, which is basically gpt image 2.0 and above
none of which is to dispute the original question which is perhaps "why are image models so smol vs LLMs" - which i agree with and wish there were funding for scaling them - but the abject failure of sora indicates that perhaps our data isn't nearly good enough (again pointing to another thing - much easier to pretrain and posttrain LLMs given quantity of text data)
1 reply →
> since an LLM should be able to describer [sic] an image losslessly
Any lossless text description of a given image would just be a structured text output of the exact pixel values, or an encoding that losslessly exploits regularities. Anything less is categorically lossy. An ordinary semantic description of a reasonably complex image will generally be lossy. A short, lossless text description would need to exploit regularities in the image, such as a solid-color image, or a basic pattern or sequence which is easily reproducible, such as a precisely specified checkerboard image.
And being able to represent images losslessly in text simply establishes text’s capacity to encode information. It doesn't establish an LLM’s holistic understanding of images. There is heavy lifting in the phrase “images are a subset.” An image can be almost any conceivable perceptual arrangement of data. The bounds are not clear. Maybe what you meant to suggest was that images of our natural world contain exploitable regularities.
> Natural evolution has also had a much easier time coming up with vision models (everything with eyeballs) than complex language models (as far as we know only a smallish family of monkeys).
It isn't appropriate to explain complex biological systems in terms of today's technologies. This is something people have been doing for a long time. The mind is like a a book, a loom, a camera, an engine, a computer, an LLM/vision model. Etc. The cognitive abilities required for an organism to meaningfully interpret its vision can possibly largely be the same cognitive abilities underlying speech, and all that is missing for a brain to exploit this might be sufficiently advanced vocal cords or manual dexterity; nothing to do with the brain itself, but its auxiliary organs.