Comment by gallerdude

3 months ago

For coding, I like the Aider polyglot benchmark, since it covers multiple programming languages.

Gemini 2.5 Pro got 72.9%

o3 high gets 81.3%, o4-mini high gets 68.9%

6 comments

gallerdude

Isn't it easy to train on the specific Exercism exercises that this benchmark uses?

where do you find those o3 high numbers? https://aider.chat/docs/leaderboards/ currently has gemini 2.5 pro as the leader at, as you say, 72.9%.

re-thc 3 months ago
It's in the OpenAI article post (OP) i.e. OpenAI ran Aider themselves.
- vessenes 3 months ago
  
  Update: the leaderboard has o3 high + 4o tops of the charts now with 82.7%. This is a) amazing b) 20x more expensive than Gemini.

It was a good benchmark until it entered the training set.

thanks