It’s very likely they all add svg generation into the training data. It’s part of the reason it’s no longer a good benchmark (unless you need to generate SVGs).
It's about more than just training data. Every model since GPT-2 had SVGs in the training data, because they all used a scrape of the Web and the Web is full of SVGs. What's unique about Gemini is that they actively worked to get better results for their SVGs - maybe RLHF, maybe RLVF of some sort.
Its giraffe / grandfather clock one is pretty bad... (two necks? wearing a suit?)
Weird as well, it's clearly pulled out some 1884 patent on clock designs, and a quick ddg/google doesn't show it as anything to do with grandfather clocks.
Google Gemini are the only team who have openly had staff deliberately spend time on SVG performance: https://twitter.com/sunjiao123sun_/status/202455551655137292...
See also this Jeff Dean tweet showing off their animals-in-vehicles abilities: https://twitter.com/JeffDean/status/2024525132266688757
It’s very likely they all add svg generation into the training data. It’s part of the reason it’s no longer a good benchmark (unless you need to generate SVGs).
It's about more than just training data. Every model since GPT-2 had SVGs in the training data, because they all used a scrape of the Web and the Web is full of SVGs. What's unique about Gemini is that they actively worked to get better results for their SVGs - maybe RLHF, maybe RLVF of some sort.
Its giraffe / grandfather clock one is pretty bad... (two necks? wearing a suit?)
Weird as well, it's clearly pulled out some 1884 patent on clock designs, and a quick ddg/google doesn't show it as anything to do with grandfather clocks.
Gemini 3.8 Flash results are often not very coherent but it does put a lot of shading and details to hide the fact.