Comment by wongarsu
4 hours ago
One fun example: "Ruskin Bond was born on May 19, 1934, in Kasauli, Himachal Pradesh, India". Opus and Gemini believe this to be true, GPT 5.4 believes it's false, Sonar thinks it's mostly true. Disagreement value of 3, you can't disagree more than some models thinking it's true, some thinking it's false
But my impression from 2 minutes on Wikipedia is that the most likely disagreement is on the "Himachal Pradesh, India" part. The guy was born on that date, in that town. But while the town is today in the state of Himachal Pradesh in India, that was not true in 1934. When he was born, the city was in the Punjab States Agency of the British Raj.
So was he born in Himachal Pradesh, India or not? I find both True and False equally defensible here
https://lite.datasette.io/?csv=https%3A%2F%2Fstatic.simonwil...
How can someone be born in a state that does not yet exist? The statement has the year in it clearly demonstrating the contradiction. One can't be born in the Soviet Union in 1995 or in Tsarist Russia in 1950.
There's lots of things like this where if you ask a human, the answer will change depending on what's convention in their subculture.