Comment by stared
3 hours ago
In the spirit of ARC-AGI-3-like challenges, we just tested if frontier AI models are able to solve a lovely puzzle game, Baba Is You: https://quesma.com/blog/baba-is-bench/
A year ago, Sonnet 4 barely solved the first level. Now, both Fable 5 and GPT-5.6 Sol beat the first two stages. GPT 5.2 is slow, but efficient, while Gemini 3.1 Pro and 3.5 Flash struggle.
I'm wondering what's up with the release of Gemini 3.5 Pro, they keep postponing it. For a while, Google was doing pretty well with their releases.
Heh, probably something like this.
Works fast - tells people how to overthrow the government.
Follows all rules and conventions Google wants - says corporate speak without actually accomplishing anything.
Can actually do complicated things- apt to tell the user to fuck off and do the hard work themselves.
Training models seems more akin to raising a kid than a computer application.
FWIW: "Baba Is You" is 7 years old and heralded as one of the greatest puzzle games of all times, with guides and solutions shared all over the internet. How to beat this game is 100% in the training set.
It was our original assumption. Yet, we went through trajectories and agents did not recall solution. It is with a sharp contrast with task for which agents magically generate solution, e.g. https://openai.com/index/why-we-no-longer-evaluate-swe-bench....
In a few instances (we covered it in Caveats) Gemini 3.5 Flash "knew" which level it was, but misremembered, and went with a wrong solution.
It turns out knowledge and ability are not the same thing. We should test both.