Comment by packageman

I worked with image analytics years ago. In 2024 and cynical about LLM image capability, I tested different models to create "Two female octopi in a bar each having their own drink. One is wearing a yellow hat. The other is wearing a red hat. They have no human features." It went badly I think because the training visual training data associated with the word "female" was overwhelmingly biased in quantity and obviated the 'no human features' instruction even after iterated instructions to remove.

So I tried "Make an image of a Djibouti cab with a camel sitting in the passenger seat. Give the camel no human anatomical features." Still bad.

Two years later the graphics and perspectives of the tools are so much better. But that isn't the big story. What really stands out is the models are no longer adding human female anatomies to camels and octopi. The gates and filtering based on instructions have matured enough so that the pelican/bicycle deductive reasoning puzzle is less problematic. But until an LLM anticipates something like impressionism from Parisian artists rebelling against the rules of the French Academy, continue to reserve a place for human artistry.

tl;dr simonw, your spot-check nailed it just as well as any extensive methodology. LLM's no longer need to cheat this part of the test (better to hack the question than the tool).