Comment by 5kyn3t
10 hours ago
For what do you guys use vision in those models? surveillance is the obvious use case... but are there some "nicer" ways to use it?
10 hours ago
For what do you guys use vision in those models? surveillance is the obvious use case... but are there some "nicer" ways to use it?
The obvious use case, especially on HN, is frontend dev of any kind at all. The second most obvious one is OCR of paper documents.
Frontend Dev? I do not really understand. do you let the models analyze the webpages you are working on? or for testing?
LLMs are not great at aligning stuff on first try, they are however very good at taking screenshots and fixing their mistakes. Claude Design also does this all the time, as does regular Claude in the web UI if you tell it to make a powerpoint presentation
I really missed this feature when I had DeepSeek code a small game for fun. When writing UI and rendering code it could execute the game and get screenshots back, but then had to rely on my feedback on what had gone wrong. Models with vision can do much better here, finding more issues on their own
Standard flow with a vision model in OMP is to write the front end code, fire up the server, fire up a headless browser and then take screenshots and examine and iterate. Works great. When I'm using DeepSeek V4 Flash, it always reminds me instead that I have to validate manually by loading up the page.
QA of course. You hook up your agent with CDP access to live product + let it screenshot and look into result. Also you could hook agent with CDP access to Figma to read/write, there a vision model is very useful as well.
It closes the development loop. Without it a model can't check if the stuff it made actually visually renders like it's supposed to. It can only guess/assume.
but for OCR there are much better suited models, I use mlx-community/PaddleOCR-VL-8bit
Sometimes you intentionally want to verbatim keep "mistakes", sometimes you don't and want them to be "fixed". OCR-only models tend to only do one of those two, in VLM cases often the latter. With multi-modal LLMs you can just tell them (adherence of course needing evals/differs per model).
Any kind of spatial/graphical task is likely going to go better with a vision-capable model. Feed it a napkin-sketch of what your app should look like. Have it verify screenshots of the UI it just built. All of these one-shot-a-video-game evaluations that have suddenly become popular only work if the model can interpret screenshots...
My product is connecting employers and workers with conversational agents. They love to communicate with images — CVs, documents, photos of worksites. Even CV-as-photo or offer-as-photo format is very popular. My daily driver Deepseek Flash can't see those photos. So I use image models to let agents understand the context.
I use research agents to attribute methane emissions plumes detected by satellites to oil and gas infrastructure on the ground, using a pre-baked database of geospatial data and web research.
Had a tool that called out from DeepSeek to Gemini 3.5 Flash for viewing the spatial features in the context of high-resolution satellite imagery of each site, but will be trialling this model for the whole thing now.
https://stencil.so/blog/snapcompact - some agents (notably oh my pi, i forget which others) come with snapcompact as a primary means of compaction. Take the entire context, stick it in a small font in a PNG, and vision capable models can summarize and pull out the most useful information in many fewer vision tokens than the original context used.
I've not used it myself, but it's there.
Having vision is very handy for getting it to make plots/figures with matplotlib. A model with vision can be much more autonomous with catching visual glitches/misalignments and correcting itself.
Also used it for 3d printer control once, had it diagnosing issues, calibrating my Tradrack MMU and canceling failed prints autonomously from a couple of cameras placed around the printer.
Allowing it to analyse a system under test (usually in an emulator, web browser, Electronic app container, etc. - something that can be reasonable captured).
It makes running much, much longer feedback loops possible. Although you can mix and match non-vision and vision models simply by invoking a vision model when you need one, as I like to use non-vision models like glm-5.3.
Going straight to surveillance and unable to think "nicer" ways... is strange.
1. process graphs and charts
2. process handwritten math formula, also chinese characters writings
3. process design sketch and wireframe
4. process scanned documents
... etc
in fact these transformer models currently suck for surveillance, too slow and expensive. There are already faster and better facial/gait/object recognition models out there.
In the feedback loop when working on anything UI or graphical output related.
I've been working on an agentic graphic design tool, so vision is quite useful for having the model check its own work. I'm already seeing improvements with this model vs the text-only one.
No one’s mentioned robots, so… robots. VLA models, etc.
Generating alt text for images in social media posts.
frontend design work, game development
when i am learning i draw what i undestand in a picture and ask ai to correct me. i want ai to watch over me while i am learning.
this is such good way to learn something for me.
[dead]