It's also the only model that generates accurate translation and localization. No other frontier model comes close. Although Gemini's coding capabilities are subpar, its natural language processing is top-tier.
I’m curious how you guys keep track of each model’s coding capabilities. The landscape keeps changing. I don’t suppose you benchmark all frontier models every other month, right?
That might depend on whether you are translating fiction or nonfiction.
Anecdotally I'd rate Gemini behind Claude and OpenAI models at fiction and I can't find any benchmarks showing Gemini is the clear winner at this task.
I found that it's shockingly good with R. (the only language I know and can correct for)
I doubt they even intended it to be, but it seems like I kept going from resorting to 3.5-3.8 (over time) to realizing that Claude and GPT, while great at Python, will make rudimentary mistakes with R; even when they compose giant complicated R code.
I'm guessing this is partly because of Gemini's world knowledge. I tried asking the model multiple internet humor and memes and it answered correctly around 80% of the time
Agreed. My impression is that the more verbose output of sol, astra etc is that it helps it steer itself on long running tasks (but is worse for the human user to read)
Yes I've noticed there's also this drive to implement and start talking about how it would write specific portions of code in response to design/trade off questions. I have to prompt Sol/Astra almost every time with a note that I am not looking for implementation advice since I mostly use them as a rubber duck in the design phase
Is this a "pro" thing? I have totally no idea what I'm talking to, so actually I'm thinking of stopping my plan. Gemini and Claude are much more clear about it.
Anyway, I like the speed at which Gemini responds so indeed for simple things it is preferable.
For me (Plus plan, iOS app), Astra only shows up under the Work tab.
I’ve been using Work for all my queries, since it seems to just be the same interface as Chat but with more features. I don’t understand why they’re two separate things.
For rabbit holes, how do you get Gemini to do any research before answering? I've very recently had it hallucinate on me like it's 2023, and that was on Pro/Thinking, as far as I remember.
Is Google still chasing frontier? Seems like they haven't had a "Pro" model in forever. I think a good niche for them would be right where they are now.
They are. They were supposed to release 3.5 Pro over the summer but haven't because of persistent architectural/technical issues allegedly. Which is better than releasing it in that state imo.
Their AI leadership team has taken some hits recently too, in the form of departures. I believe when they get their bearings they will be competitive again. 3.8 Flash has been a great model for me.
"couch all their agreements with caveats and provisos."
When you're a ChatGPT Projects or Claude Projects user, those caveats and provisos are your worst enemy because they'll change caveats into hard rules (either for the session or committed to memories) and you end up in absolute hell having to make it investigate to figure out why it can no longer produce anything but read-only pre-check code that never actually does anything but keeps performing stupid safety checks.
It's entirely possible that in their testing of newer models, the whole problem is that even if it's doing better in benchmarks, maybe it's insufferable to work with, thus they're not releasing it.
I was surprised when (finally) trying out Claude how much I preferred Gemini's way of communicating. I wont argue Claude is better at coding, but for knowledge work, I had to dig through Claude output to find what I actually wanted. At times, it even felt borderline incomprehensible.
Just today I had Sonnet 5 generate this (asking about always-on display in the iPhone e-versions):
> This mirrors how Apple has always segmented Pro vs. non-Pro iPhones: base models got LTPS panels while Pro models got LTPO, and only with the mainline iPhone 17/17 Plus did that gap close the standard versions previously lacked the smoother 120Hz ProMotion technology and the always-on display feature, unlike the Pro models — the 17e is the one model line still using the older, cheaper panel.
(emphasis mine)
I mean, I can guess what it is trying to say, but who RL'd this nonsense?
Fable 5.1 even today inundates prose with "it's not this, it's that" type of garbage. I had to rewrite two paragraphs from a generic class announcement which I was planning to post on the LMS. Not sure where that productivity gain is that everyone is talking about.
I couldn't even stand 4.6 by the end. 4.5 was very usable, 4.6 started alright and somehow got more annoying. 4.7 made me quit my subscription and ditch their services entirely - it's an honourary member of my quite short "Coworkers I'd like to throttle if I didn't work remotely" list. Infuriating to instruct or communicate with.
Not only that, Opus likes to invent arcane jargons. And the longer the task runs, the harder it is to understand its output. The conversation will be filled with uncommon word choices and awkward sentence structure.
The 4.5 models were the last Anthropic models that were actually pleasant to read. Especially Sonnet 4.5 was really fun to use for non-coding purposes. I liked having it generate silly short stories while waiting for deployments to finish. Haiku 4.5 still has a small bit of that charm, but 4.6 and later models are all unbearable unless you want a simple 'this is the answer' to a question.
I actually did (was going to travel internationally), and it wasn't as useful as you'd think. I would be talking to someone, and in the background someone else would be talking, and it would translate both people.
Only worked in a 1:1 in a quiet place. Still, can't complain for free.
As sister comments have pointed out, I find that it's very much dependent on the phone you're using. If it can't even show me a waveform then I don't think it's going to be able to tease out any sort of speech
Not the same, but related: Gemini is great for querying text in another language, it provides really cogent, useful responses with just enough source language quotes to be able to reference the source text effectively.
I can’t agree more! It feels really smooth to drive, kind of buttery compared to other frontier experiences, at least in antigravity 2.0 or whatever. I’ve been doing some web app coding with it and I’m happy with the results.
Yeah, I set it to “warm: less”, “enthusiastic: less” and “emoji: less” and it was much more bearable than I remembered it being before. Although it does love to “separate” questions when it thinks.
We have an agentic system that produces insights for end users, and runs most of its work on DeepSeek v4.1 Flash but as an output stage transforms the resulting text through Gemini 3.8 Flash for readability, and it works.
On my TODO is try and run all of the analysis pipeline in dense "machine speak" to save on tokens and just let Gemini sort it out at the end.
Wondering if people have managed to have Gemini in-front of other models like claude/codex models and only interact with that. Having Gemini act as a pure human/llm translator.
BTW, as kind of a follow-up to this, I think the most important finding to report--for those who only use one model, as many people seem to--is just how much more often Claude seems to be extremely confidently (and even insufferably) wrong than any of the other three big models (all of which I use quite often... yet I only pull out Claude when I've given up hope in a problem and are looking for out-of-the-box brainstorming).
And like, it does this despite it speaking in extremely dense math, which both makes it sound correct and requires a lot more effort to prove when it is wrong... yet, it isn't actually correct more often, and so that time sink just isn't worth the benefit. I then think many people--including people who can speak math (as can I)--just stop bothering to check everything, as if you come across a human who speaks like this it probably does correlate with slow and careful thought that helps prevent errors.
Instead, Claude has the mistake rate of a somewhat accelerated beginner impossibly combined with the language of an expert professor; and we as humans just aren't good at that combination: it becomes very dangerous and makes it take longer to spot its egregious mistakes and trained-in biases. If you have to use Claude, I thereby claim you really need to have a team of not-Claudes to help insulate you from this, and Gemini (while being a bit senile) is a lot more collaborative and approaches problems in ways that makes it harder to get tricked.
(To translate this into more of an engineering analogy: Claude always feels to me like the engineer who put more effort into learning how to program in functional languages than into how to actually develop working code, and then confidently presents you answers in Haskell or Lisp that never quite work. To find their errors is then very costly. In contrast, Gemini feels more like a Java or Go developer who knows they are a cog... that's helpful! <- Which maybe just goes to show that AI has finally turned me into a manager, omg.)
It's also the only model that generates accurate translation and localization. No other frontier model comes close. Although Gemini's coding capabilities are subpar, its natural language processing is top-tier.
I’m curious how you guys keep track of each model’s coding capabilities. The landscape keeps changing. I don’t suppose you benchmark all frontier models every other month, right?
I use them. Daily. Gemini hasn’t been a contender by comparison for a long time.
13 replies →
That might depend on whether you are translating fiction or nonfiction.
Anecdotally I'd rate Gemini behind Claude and OpenAI models at fiction and I can't find any benchmarks showing Gemini is the clear winner at this task.
I found that it's shockingly good with R. (the only language I know and can correct for)
I doubt they even intended it to be, but it seems like I kept going from resorting to 3.5-3.8 (over time) to realizing that Claude and GPT, while great at Python, will make rudimentary mistakes with R; even when they compose giant complicated R code.
I'm guessing this is partly because of Gemini's world knowledge. I tried asking the model multiple internet humor and memes and it answered correctly around 80% of the time
For heavyweight work I have been using Astra, but for rabbit holes and brain storming Gemini is far more enjoyable to interact with.
I'm worried in their push to catch up on the SOTA front, it's going to lose that natural sounding touch it currently has.
Agreed. My impression is that the more verbose output of sol, astra etc is that it helps it steer itself on long running tasks (but is worse for the human user to read)
Yes I've noticed there's also this drive to implement and start talking about how it would write specific portions of code in response to design/trade off questions. I have to prompt Sol/Astra almost every time with a note that I am not looking for implementation advice since I mostly use them as a rubber duck in the design phase
Yes, when post training models for long tasks this happens gradually. It is not easy to prevent it as such.
How do you know you're using Astra?
My ChatGPT env only says "low", "medium", "high".
Is this a "pro" thing? I have totally no idea what I'm talking to, so actually I'm thinking of stopping my plan. Gemini and Claude are much more clear about it.
Anyway, I like the speed at which Gemini responds so indeed for simple things it is preferable.
For me (Plus plan, iOS app), Astra only shows up under the Work tab.
I’ve been using Work for all my queries, since it seems to just be the same interface as Chat but with more features. I don’t understand why they’re two separate things.
2 replies →
For rabbit holes, how do you get Gemini to do any research before answering? I've very recently had it hallucinate on me like it's 2023, and that was on Pro/Thinking, as far as I remember.
Is Google still chasing frontier? Seems like they haven't had a "Pro" model in forever. I think a good niche for them would be right where they are now.
They are. They were supposed to release 3.5 Pro over the summer but haven't because of persistent architectural/technical issues allegedly. Which is better than releasing it in that state imo.
Their AI leadership team has taken some hits recently too, in the form of departures. I believe when they get their bearings they will be competitive again. 3.8 Flash has been a great model for me.
These are native speech-to-speech models, so I think they've decided not to do that anymore.
Mostly because it answers quickly and is more agreeable (too agreeable at times).
Meanwhile Claude and Astra like to couch all their agreements with caveats and provisos.
“caveats and provisos” makes me think of Robin Williams’ genie imitating William F. Buckley Jr.
1 reply →
> Meanwhile Claude and Astra like to couch all their agreements with caveats and provisos.
Sometimes that's what being smart sounds like.
7 replies →
"couch all their agreements with caveats and provisos."
When you're a ChatGPT Projects or Claude Projects user, those caveats and provisos are your worst enemy because they'll change caveats into hard rules (either for the session or committed to memories) and you end up in absolute hell having to make it investigate to figure out why it can no longer produce anything but read-only pre-check code that never actually does anything but keeps performing stupid safety checks.
2 replies →
Yea. I have asked it to verify my ideas with experiments sometimes. And it cheats and warps the results so that the results are reached
It's entirely possible that in their testing of newer models, the whole problem is that even if it's doing better in benchmarks, maybe it's insufferable to work with, thus they're not releasing it.
Good news then, I don't think they're in a hurry to catch up to SOTA.
I was surprised when (finally) trying out Claude how much I preferred Gemini's way of communicating. I wont argue Claude is better at coding, but for knowledge work, I had to dig through Claude output to find what I actually wanted. At times, it even felt borderline incomprehensible.
Just today I had Sonnet 5 generate this (asking about always-on display in the iPhone e-versions):
> This mirrors how Apple has always segmented Pro vs. non-Pro iPhones: base models got LTPS panels while Pro models got LTPO, and only with the mainline iPhone 17/17 Plus did that gap close the standard versions previously lacked the smoother 120Hz ProMotion technology and the always-on display feature, unlike the Pro models — the 17e is the one model line still using the older, cheaper panel.
(emphasis mine)
I mean, I can guess what it is trying to say, but who RL'd this nonsense?
Fable 5.1 even today inundates prose with "it's not this, it's that" type of garbage. I had to rewrite two paragraphs from a generic class announcement which I was planning to post on the LMS. Not sure where that productivity gain is that everyone is talking about.
Opus specifically talks as if having a stroke. 4.6 was the last version that was pleasant to work with
I couldn't even stand 4.6 by the end. 4.5 was very usable, 4.6 started alright and somehow got more annoying. 4.7 made me quit my subscription and ditch their services entirely - it's an honourary member of my quite short "Coworkers I'd like to throttle if I didn't work remotely" list. Infuriating to instruct or communicate with.
I work for Google and we have the choice between Gemini models and Opus. Opus is slightly better than Gemini flash but I find the style unbearable.
It reminds me of a pedantic grad student.
Not only that, Opus likes to invent arcane jargons. And the longer the task runs, the harder it is to understand its output. The conversation will be filled with uncommon word choices and awkward sentence structure.
The 4.5 models were the last Anthropic models that were actually pleasant to read. Especially Sonnet 4.5 was really fun to use for non-coding purposes. I liked having it generate silly short stories while waiting for deployments to finish. Haiku 4.5 still has a small bit of that charm, but 4.6 and later models are all unbearable unless you want a simple 'this is the answer' to a question.
Try Gemini live in a multi lingual environment. It can pick out speakers and live translate to you. Truly underrated for its capabilities.
I actually did (was going to travel internationally), and it wasn't as useful as you'd think. I would be talking to someone, and in the background someone else would be talking, and it would translate both people.
Only worked in a 1:1 in a quiet place. Still, can't complain for free.
As sister comments have pointed out, I find that it's very much dependent on the phone you're using. If it can't even show me a waveform then I don't think it's going to be able to tease out any sort of speech
3 replies →
"Only worked in a 1:1 in a quiet place"
well then its not model problem
1 reply →
Not the same, but related: Gemini is great for querying text in another language, it provides really cogent, useful responses with just enough source language quotes to be able to reference the source text effectively.
I can’t agree more! It feels really smooth to drive, kind of buttery compared to other frontier experiences, at least in antigravity 2.0 or whatever. I’ve been doing some web app coding with it and I’m happy with the results.
Chatgpt doesn’t seem so bad lately. At least as a Claude refugee.
Yeah, I set it to “warm: less”, “enthusiastic: less” and “emoji: less” and it was much more bearable than I remembered it being before. Although it does love to “separate” questions when it thinks.
I find its style the most sycophantic and annoying personally.
In my experience, with minimum prompting, deepseek also generates very decent text.
Not at all in mine. Deepseek has some of the worst prose of the close to frontier models in my opinion.
Does anyone know if there is a dedicated model which makes Claude output nore human readable and less slop?
Lately it became load-bearingly-reality-difficult to not only read, but to comprehend the Claude output
I find Astra's prose very good, too. I have been using it to rewrite all my LLM-generated docs as of lately.
It also doesn't anthropomorphise itself, like at all.
We have an agentic system that produces insights for end users, and runs most of its work on DeepSeek v4.1 Flash but as an output stage transforms the resulting text through Gemini 3.8 Flash for readability, and it works.
On my TODO is try and run all of the analysis pipeline in dense "machine speak" to save on tokens and just let Gemini sort it out at the end.
Wondering if people have managed to have Gemini in-front of other models like claude/codex models and only interact with that. Having Gemini act as a pure human/llm translator.
Somebody shared this a few days ago: https://github.com/adnanakil/nobuzz
Not in the principled sense you mean but I have in fact recently started having Gemini explain to me what Claude is talking to me about, lol.
BTW, as kind of a follow-up to this, I think the most important finding to report--for those who only use one model, as many people seem to--is just how much more often Claude seems to be extremely confidently (and even insufferably) wrong than any of the other three big models (all of which I use quite often... yet I only pull out Claude when I've given up hope in a problem and are looking for out-of-the-box brainstorming).
And like, it does this despite it speaking in extremely dense math, which both makes it sound correct and requires a lot more effort to prove when it is wrong... yet, it isn't actually correct more often, and so that time sink just isn't worth the benefit. I then think many people--including people who can speak math (as can I)--just stop bothering to check everything, as if you come across a human who speaks like this it probably does correlate with slow and careful thought that helps prevent errors.
Instead, Claude has the mistake rate of a somewhat accelerated beginner impossibly combined with the language of an expert professor; and we as humans just aren't good at that combination: it becomes very dangerous and makes it take longer to spot its egregious mistakes and trained-in biases. If you have to use Claude, I thereby claim you really need to have a team of not-Claudes to help insulate you from this, and Gemini (while being a bit senile) is a lot more collaborative and approaches problems in ways that makes it harder to get tricked.
(To translate this into more of an engineering analogy: Claude always feels to me like the engineer who put more effort into learning how to program in functional languages than into how to actually develop working code, and then confidently presents you answers in Haskell or Lisp that never quite work. To find their errors is then very costly. In contrast, Gemini feels more like a Java or Go developer who knows they are a cog... that's helpful! <- Which maybe just goes to show that AI has finally turned me into a manager, omg.)
1 reply →
[flagged]
Terrible for code, amazing for prose
I've set my documentation sub agent to Gemini and my code agent to Luna