Comment by noname120
2 days ago
Given that both Gemini 3.5 Flash (high) and Gemini 3 Flash Preview (medium) beat GPT-5.6 Sol (high) for correctness and score in your benchmarks I don’t trust them at all. The rest of the ranking also doesn’t make sense, like GPT-5.3-Codex (medium) performs better than Claude Opus 4.8 (medium) yeah sure
It's because the benchmark is not coding-only.
Gemini models tend to have most knowledge for most domains, and are one of the most intelligent overall. You can check other benchmarks too, on specific categories, those models still beat other SOTA models.
Regarding Opus, Anthropic models often fail to follow instructions, formatting requirements or simply refuse to answer questions (i.e. Fable).
The issue with Gemini models is that they are not as good as using tools or go into weird failure modes when coding or trying to extract/generate specific data. They work amazing, until they don't...
100% agree with your statement that gemini models have most knowledge in domains. They shine in so many generic topics
One example where the order seems correct, is this SVG generation test:
https://aibenchy.com/showcase/?q=Gemini+3.5%2Cgpt+5.6%2C+5.3...
You can see that most Gemini 3.5 generations are more correct than 5.6 Sol (the net is in the middle of the table, hamster seems reasonable and not deformed, etc.)
It doesn't seem correct at all though? the supposedly best one isn't even the best gemini flash output (the medium one looks better than the high one)