Comment by fzysingularity

Exactly! We experimented with a whole bunch of video encoding techniques for LLMs here: https://vlm-run.github.io/mm/encoders/#video