Comment by api
1 day ago
It kind of blows me away how “unreasonably effective” these small image models seem for their size.
Meanwhile a decent LLM that can do tool use and code and reason tends to weigh in at a minimum of 30B parameters, and you need over 100B to get good performance.
Do LLMs just contain more information or are diffusion models more efficient at compression?
I saw some work on language diffusion models a while back but I think they tend to have problems with reasoning versus transformer type models.
These models are, in a sense, distillations of language models. They wouldn't be as good without a pre-trained language model, so it's unfair to compare them.
Image models are essentially a mapping of text embedding space -> image. So the clearer and cleaner the text embedding space is the better model you get as you need less weights trying to interpret a text prompt, recent image models even use multimodal versions of LLMs (Flux uses Qwen3 VL) so all the visual separation was done beforehand making the model even more efficient.
edit: Maybe a better analogy is that the image models are only half the model in a encoder-decoder sense, where the LLM is a pretrained frozen encoder and the image model is the decoder.
To make a rather speculative argument - since an LLM should be able to describer an image losslessly, images are a subset of what an LLM needs to understand. It is possible that the statistical model being learned for images is just simpler.
The get a proper LLM they have to model all of human knowledge regarding patterns, physics, biology, social interaction, skills, history, etc, etc. For images they have to recall what a dog looks like in a still photo. The image models aren't even necessarily good at a wide range of novel situations; usually they get used for stock scenes.
Natural evolution has also had a much easier time coming up with vision models (everything with eyeballs) than complex language models (as far as we know only a smallish family of monkeys).
this is the right answer. ultimately text and general llm usage incl coding is far more internally complex than just memorizing the looks and composition of concepts placed in an image. dont forget that most of our reasoning benchmarks basically require the llm to have a search engine and calculator in its own weights.
These diffusion models are not just memorizing the looks and composition of concepts. These models are increasingly capable of representing advanced world knowledge. Things like shadows require a comprehensive, if indirect, understanding of physical light modeling. Things like weight/size estimation, etc., which are all important to a cohesive generated image of natural phenomenon, require an adequate understanding of physics.
Some of this understanding is in the input embeddings themselves, embedding models do a lot of heavy lifting behind image generation models.
I posit that you can theoretically achieve the same capability in a model trained on the right sequence of "images" as you can purely text; both are turned into embeddings and can ultimately convey the same exact relationships. I don't think it's very useful to try and create delineations between text and image inference, especially as more models move towards native multimodality.
2 replies →
> since an LLM should be able to describer [sic] an image losslessly
Any lossless text description of a given image would just be a structured text output of the exact pixel values, or an encoding that losslessly exploits regularities. Anything less is categorically lossy. An ordinary semantic description of a reasonably complex image will generally be lossy. A short, lossless text description would need to exploit regularities in the image, such as a solid-color image, or a basic pattern or sequence which is easily reproducible, such as a precisely specified checkerboard image.
And being able to represent images losslessly in text simply establishes text’s capacity to encode information. It doesn't establish an LLM’s holistic understanding of images. There is heavy lifting in the phrase “images are a subset.” An image can be almost any conceivable perceptual arrangement of data. The bounds are not clear. Maybe what you meant to suggest was that images of our natural world contain exploitable regularities.
> Natural evolution has also had a much easier time coming up with vision models (everything with eyeballs) than complex language models (as far as we know only a smallish family of monkeys).
It isn't appropriate to explain complex biological systems in terms of today's technologies. This is something people have been doing for a long time. The mind is like a a book, a loom, a camera, an engine, a computer, an LLM/vision model. Etc. The cognitive abilities required for an organism to meaningfully interpret its vision can possibly largely be the same cognitive abilities underlying speech, and all that is missing for a brain to exploit this might be sufficiently advanced vocal cords or manual dexterity; nothing to do with the brain itself, but its auxiliary organs.
At first, I considered that diffusion models struggle with reasoning because they underweight 'prior beliefs', ie the propositions by which an argument advances.
Then I considered whether the universe of imagery is simply smaller than the universe of ideas—which must, itself, include the universe of images, too.
But finally, I'm thinking maybe images are just harder to benchmark.
Or maybe they're _not_ actually worth 1,000 words.
When it comes to reasoning specifically, it's a far simpler explanation IMO. Modern RLVR techniques like GRPO rely on the model understanding the likelihood of the generated phrase. This is really easy to compute for LLMs that are trained with next-word prediction. Diffusion models have a harder time accessing this likelihood value (i.e., you can approximate it but the approximation isn't good). Hence, reasoning doesn't work (yet) for diffusion models. Though, alternative approaches are currently being tried out in academia; so someone might crack it!
What is meant by "understanding" the likelihood?
Diffusion models, surely, are "trained with next-[pixel] prediction", and that should suffice, no?
They can make good images but they all look the same. Guess it's not hard to replicate a small section of what exists.
When you say "they", what are you referring to? Image generation models? LLMs? All of them or older/more recent ones?
When you say "look the same", what do you mean?
I understand this was just a casual one-off comment, but as presented I'm having a hard time understanding what you're trying to convey and if you have appropriate domain knowledge: Your "looks the same" and my "looks the same" might be vastly different, especially depending on context.
[flagged]