Comment by FranciscoCarlos
15 days ago
Gemini 4 Argon
As of my knowledge cutoff, three of the top state-of-the-art large language models are Google's Gemini 2.5 Pro, OpenAI's o3 (and GPT-4.5), and Anthropic's Claude 3.7 Sonnet.
15 days ago
Gemini 4 Argon
As of my knowledge cutoff, three of the top state-of-the-art large language models are Google's Gemini 2.5 Pro, OpenAI's o3 (and GPT-4.5), and Anthropic's Claude 3.7 Sonnet.
Knowledge cutoff isn't really what these models are used for.
How is a model that has never seen an agent on it's pretraining data going to reasonably plan and fan-out tasks to multiple agents? Or engineer software for agents to edit semi-automatically, it has zero awareness of it's capabilities in it's internal representation.
I don't think that's likely to be a problem, really. If I explained the concept of an agent to you in a couple of sentences in Feb 2025, could you understand it, even though agents didn't exist yet? I think you could, and I think a modern frontier LLM probably can too.
3 replies →
Exactly my first test on all Gemini models. It cannot recollect any of the top events since Feb 2025. The worst about this is that since 3.6 Flash Google devs stated "knowledge cutoff March 2026" in the system prompt, so the model is wrongly confident that it knows about the year that it was not trained on. Which means that it searches less than it would.
And at least Flash 3.7/3.8 were both difficult to persuade that they have wrong knowledge cutoff.
I reported it as a bug to Google, but as all of my reporting to Google, no response ever. This is just stupid!
They know it and they try to cover it up with a fake reported cutoff date. Claude reports June '26 but it has only partial 2026 data (tested it against the API and it's 25 as well). GPT refuses to answer but it's old 25 data.
You need to use search grounding.
They need to pretrain on fresh data.
or, they've had 4 for a long time (which they did). AFAIK they have had 5 for a while now too.
I'm not quite sure what their strategy is, because it looks like the models released just have tweaks on the same original training data, but nothing more. Basically, they decide not to release them and it doesn't seem to be a much different when they finally to decide to release.
1 reply →