Comment by SkalskiP
20 hours ago
Hi, I’m the author of this blog post. I wrote it about 4 weeks ago, and the VLM world is moving so fast that it’s already kinda outdated. I think Gemini 3.7 Flash might be a better choice now, especially when you factor in the price.
Here’s a comparison of the best low-cost models I put together last week. What’s crazy is that Gemini 3.7 Flash is now 50% off on OpenRouter, and this chart doesn’t even account for that discount. https://x.com/skalskip92/status/2088032652301304121?s=20
Curious why you didn't try Gemini 3 pro? That is the model I've been using for OCR entry of handwritten datasheets (JPGS of datasheets, structured JSON output). At my scale, the cost of 3 pro is basically not an issue, but if there are improvements in quality, I'd definitely be willing to explore other models
In my experience starting with Gemini 2.5 Pro, moving to 3 and 3.1, 3.5 Flash, 3.6 Flash, and finally 3.7 Flash, 3.7 Flash is just as good if not better than 3 especially on high resolution mode (same token count per page as 3.1).
I run complicated, messy PDFs through these models. 2.5 Pro required a lot of kludgy hacks to get it to fully "see," but from 3.1 pro on I've removed many of them and haven't spotted problems.
3.7 Flash scores better than 3.1 pro on most benchmarks, leading me to believe that even if your OCR requires reasoning to interpret text or data, 3.7 Flash is probably going to be better.
3 Pro is quickly approaching one year old. There's almost no reason to benchmark it, especially since a new version of Gemini Pro was supposed to be released mid 2026 and hasn't seen the light of day.
That would make sense if we already knew that, for these kinds of tasks it was significantly worse. The tests that I'm aware of for these tasks show it as still performing near the top.
I think it definitely makes sense since it's still the best Google has to offer in the "pro" tier.
4 replies →
The “pro” moniker means nothing
these models aren’t successors and barely have a common ancestor, they are independently baked in the training oven and assigned a semantic version randomly by someone trying to show initiative but not trying to do on the toes of the last guy who got promoted first
So 3 pro is outdated and will likely never exit preview
The “flash” and “lite” models are the real “pro” in colloquial ideas of fleshed out and capability, at this point.
they’re better, faster and cheaper, larger context windows keeping up with the industry and more
They are smaller models, and you can tell. Small models make dumb common-sense mistakes that big models never do. This is the "smell" many talk about.
4 replies →
[dead]
At this point, VLM benchmarks should probably come with an expiration date. A four-week-old leaderboard can already be measuring a different market.
What about Gemma ?