Comment by merlindru
15 hours ago
your question may have become part of the training data with how much coverage there was around it. perhaps you should devise a new test :P
15 hours ago
your question may have become part of the training data with how much coverage there was around it. perhaps you should devise a new test :P
I suggest asking it to identify/count the number of fire hydrants, crosswalks, bridges, bicycles, cars, buses and traffic signals etc.
Pit Google against Google :D
3.1 Pro has the same Jan 2025 knowledge cutoff as the other 3 series models. So if 3.1 has it in its training data, the other ones would have as well.
The fact it's still Jan 2025 is weird to me. Have they not have a successful pretrain in over a year?
They likely won’t have a full pretrain for awhile. Just like everyone else, the name of the game now is to finetune existing models.
My job may have become part of the training data with how much coverage there is around it. Perhaps another career would be a better test of LLM capabilities.
Have you ever heard of a black swan?
Honestly at this point I have fed this image in so many times on so many models, that it also functions as a test for "Are they training on my image specifically" (they are generally, for sure, but that's along with everything else in the ocean of info people dump in).
I genuinely don't think they are. GPT-5.2 still stands by 4 legs, and OAI has been getting this image consistently for over a year. And 3.1 still fumbled with the harder prompt "How many legs does the dog have?". I needed to add the "count carefully" part to tip it off that something was amiss.
Since it did well I'll make some other "extremely far out of the norm" images to see how it fairs. A spider with 10 legs or a fish with two side fins.
Easy fix, make a new test image with six legs, and watch all the LLMs say it has five.