Comment by m_w_
11 hours ago
It's a bit disheartening to see no comparison to other models here - and I'm not sure this pushes the curve anywhere. 3.6 flash is more expensive than GLM 5.2 - but seemingly worse, although this post is really light (lite?) on details.
It seemed for a time that Google had finally gotten the ball rolling, but I'm doubting that more and more as time passes. We'll see what happens with 3.5 pro I suppose.
It’s really surprising. When Apple announced the multi-billion dollar deal with Google to power Apple Intelligence I thought great things were coming. Instead we are getting more and more bad news: delayed Pro models and AI leadership leaving. I wonder if Apple know something the rest of us don’t know or if they are already regretting their decision.
Besides Apple apparently making Siri AI model agnostic, the choice to go with Google was almost certainly for practical reasons. Google is a low-risk established player that already has a long work history with Apple. Google also isn't in an existential battle to establish themselves, Gemini still amounts to just another project at Google. There is tangible non-zero risk that either OAI or Anthropic will be gone in 5 years, or will be forced to leave Apple high and dry to save themselves. There is almost no risk Google will be in either such position. And worst case scenario, Google has incredibly deep pockets should Apple pursue a "refund."
What Apple wants out of Google is Siri that runs at 8gb ram and isn’t a horrible embarrassment that feels like a primitive markov chain. Given how good Gemma 4 is, Google can squeeze some serious performance in small models. Whether they can make bleeding edge models is irrelevant to Apple.
"Siri, please solve the Jacobian conjecture, and also set an alarm for 8am tomorrow"
Apple isn't counting on their model to be a frontier coding and cowork model. Gemini is perfectly fine for the tasks that new Siri is supposed to be doing.
As someone on the Apple beta.. the model is almost completely irrelevant to the experience. Apple has gone and done Apple things by nerfing the experience so completely that almost any model in the past year would be fine. I still reach for ChatGPT/Claude/Grok constantly instead of the AI toy that Apple calls the new Siri.
Here, my comparison of 3.6 Flash vs Sol vs Luna vs Terra: https://aibenchy.com/compare/google-gemini-3-6-flash-medium/...
How does your comparison work? It places Gemini 3.6 Flash Medium above GPT 5.6 Sol High and Fable 5 Medium, which makes me skeptical because that... would be making headlines that I'm not seeing right now.
I have created various questions/tests and put the models through the same tests.
I record whether the answers are correct, and the generation stats (costs, latencies, tokens used, etc.).
I have no idea why the Gemini models do so well.
I have recently added new tests, whose sole purpose was to find some cases on which Gemini 3 Flash fails (I don't like cherry-picking models or tests, but I also find it strange Gemini Flash models leading in accuracy). I made a more complex coding/tool-usage test, that I expected it to fail, it did fail it once locally in my debug tests, but when I finalized the test and ran the entire testing suite for all models, somehow Gemini 3 Flash still got it right...
Gemini models are REALLY intelligent (and they are actually my favorite model to use via the chat app to ask questions), but they somehow fail in real-word coding tasks where they have to modify files, check results, debug, etc.
My tests harness provides a lot of mock data, and limits the number of actions a model can choose from. I am starting to think that maybe the models are not bad, just that the coding harness are not optimized for those type of models, and Google doesn't really provide their own "Codex".
3 replies →
GLM was twice as verbose running the Artificial Analysis benchmark. So it ends up being more expensive
>verbose
GLM defaults to max effort btw
https://docs.together.ai/docs/glm-5.2-quickstart#reasoning-e...
Not really. Gemini 3.6 Flash actually cost $0.01 more per task, compared to GLM 5.2.
https://artificialanalysis.ai/models/gemini-3-6-flash
But to run the entire benchmark it cost $727 with Gemini 3.6 Flash and $925 with GLM-5.2, $198 (21.4%) less. I tend to look at the cost to run the whole index rather than the weighted average cost per task.
All the benchmarks I see put it around the capabilities of Opus 4.8 Medium or Sonnet 5 High.
As far as I can tell it's slightly better than GLM 5.2.
according to AA it's not better than GLM 5.2 and that's surprising to me
I personally take AA with a giant handful of salt.
The one thing I've found google's models to be the best at is proofreading text in non-english languages. Probably because I imagine they have the most training data for it as Google probably has the most complete archive of the internet.
> really light (lite?) on
Light. Lite is product marketing seepage.
Yeah that’s the joke :p
I've never questioned the word lite before because it's existed my whole life.. So does it make sense? Why does it exist and where does it come from? More than coming from "light".
1 reply →
Sorry, it went right over my head!
My usage of GLM 5.2 has been defined by slow throughput and flaky providers, in many (definitely not all!) applications a dumber, faster, and more consistent model makes more sense to me.
> It's a bit disheartening to see no comparison to other models here
Disheartening, but not surprising: the comparison would not be very flattering for Google.
[dead]