Comment by ricardobeat
1 day ago
The "omni" naming for a video-only model is so confusing. I was under the impression omni referred to models that can output text and audio too.
1 day ago
The "omni" naming for a video-only model is so confusing. I was under the impression omni referred to models that can output text and audio too.
To my understanding it has nothing to do with output and entirely to do with the understanding of the input together as basically one input stream. An LLM with a vision encoder or audio encoder takes the media and turns them into tokens the LLM reads and then turns into its text latents (basically).
It’s like the difference between me looking at a picture, describing it to you, and then you trying to form that picture in your mind. You might get close, but it will never be the picture I’m looking at. If I then ask you questions about the picture, you can’t answer any questions about any details that I didn’t describe. I believe that is exactly the information and understanding loss that an Omni model attempts to address, which doesn’t have anything to do with output.
Some models do also offer “Omni” output, but that is more like a feature of the model than what Omni models are by design I think.
[dead]