Comment by margorczynski
8 hours ago
If these numbers are true then OpenAI is probably done, Anthropic too. Still, it's hard to see an effective monetization method for this tech and it clearly is eating Google's main pie which is search.
8 hours ago
If these numbers are true then OpenAI is probably done, Anthropic too. Still, it's hard to see an effective monetization method for this tech and it clearly is eating Google's main pie which is search.
For SWE it is the same ranking. But if Google's $20/mo plan is comparable to the $100-200 plans for OpenAI and Anthropic, yes they are done.
But we'll have to wait a few weeks to see if the nerfed model post-release is still as good.
I have a few secret prompts to test complex reasoning capabilities of new models (in law and medicine). Gemini (2.5 pro) is by a wide margin behind Anthropic (sonnet 4.5 basic thinking) and Openai (pro model) on my own benchmark and I trust my own benchmark more than public leaderboards. So it's the other way around. Google is trying to catch up where the others are. It just doesn't seem so to some because Google undercuts prices and most people don't have own complex problems with a verified solution to test against (so they could see how bad Gemini is in reality)
This thread is about Gemini 3. It will be interesting to see your benchmark results when it's available later.
Why? These models just leapfrog each other as time advances.
One month Gemini is on top, then ChatGPT, then Anthropic. Not sure why everyone gets FOMO whenever a new version gets released.
I think google is uniquely well placed to make a profitable business out of AI: They make their own TPUs so don't have to pay ridiculous amounts of money to Nvidia, they have a great depth of talent in building models, they've got loads of data they can use for training and they've got a huge existing customer base who can buy their AI offerings.
I don't think any other company has all these ingredients.
While I don’t disagree that Google is the company you can’t bet against when it comes to AI, saying other companies are done is a stretch. If they have a significant moat then they should be at the top all the time by then which is not the case though.
8 replies →
The TPU are a key factor. They are the most mature alternative to Nvidia. Only Google cloud, Azure, and AWS enable you to rent their respective AI chips. Out of those three, google is the only one to have a frontier model. So if they have a real advantage they're not exposed to the financial shenanigans propping up neo clouds like Coreweave.
The bear case for Google was always the business side would cannibalize the AI side. AI makes search redundant which kills the golden goose
100% the reason I am long on Google. They can take their time to monetize these new costs.
Even other search competitors have not proven to be a danger to Google. There is nothing stopping that search money coming in.
Considering GPT 5 was only recently released, it's very unlikely GPT will achieve these scores in just a couple of months. If they had something this good in the oven, they'd probably left the GPT 5 name to it.
Or maybe Google just benchmaxxed and this doesn't translate at all in real world performance.
They do have unreleased Olympiad Gold-winning models that are definitely better than GPT5.
TBD if that performance generalizes to other real world tasks.
GPT 5 was released more than 3 months ago. Gemini 2.5 was released less than 8 months ago.
1 reply →
Or else it trained/overfit to the benchmarks. We won't really know until people have a chance to use it for real-world tasks.
Also, models are already pretty good but product/market fit (in terms of demonstrated economic value delivered) remains elusive outside of a couple domains. Does a model that's (say) 30% better reach an inflection point that changes that narrative, or is a more qualitative change required?
The only one it doesn't win is SWE bench which it is significantly behind Claude Sonnet. You just can't take down Sonnet.
One percentage point is not significant, neither in the colloquial nor the scientific sense[1].
[1] Binomial formula gives a confidence interval of 3.7%, using p=0.77, N=500, confidence=95%
Codex has been much better than Sonnet for me.
On what types of tasks?
1) New SOTA models come out all the time and that hasn't killed the other major AI companies. This will be no different.
2) Google's search revenue last quarter was $56 billion, a 14% increase over Q3 2024.
1) Not long ago Altman and the OpenAI CFO were openly asking for public money. None of these AI companies have actually any kind of working business plan and are just burning investor money. If the investors see there is no winning against Google (or some open Chinese model) the money will dry up.
2) I'm not suggesting this will happen overnight but especially younger people gravitate towards LLM for information search + actively use some sort of ad blocking. In the long run it doesn't look great for Google.
No, you suggested that LLMs are clearly eating google's lunch already, and there's just no evidence of that. Quite the opposite.
This may just be bad recollection from my part, but hasn't Google reported that their search business is right now the most profitable it has ever been?
They're constantly matching and exceeding each other. It's a hypercompetitive space and I would fully expect one of the others to top various benchmarks shortly after. On pretty much every leading release someone does this "everyone else is done! Shut er down" thing and it's growing pretty weird.
Having said that, OpenAI's ridiculous hype cycle has been living on borrowed time. OpenAI has zero moat, and are just one vendor in a space with many vendors, and even incredibly competent open source models by surprise Chinese entrants. Sam Altman going around acting like he's a prophet and they're the gatekeepers of the future is an act that should be super old, but somehow fools and their money continue to be parted.
This. If I had to put my money on a survivor, it would be Google because it is an established company with existing revenue modules unrelated to AI. Anthropic and OpenAI won't stand alone without external funding
I'd love to see anthropic/openai pop. back to some regular programming. the models are good enough, time to invest elsewhere