Comment by coldtea
3 hours ago
>Update: here's a better example: "Incomplete Egypt visa application forms are among the most common reasons Egyptian visa applications are rejected."
The models were split between "true" and "mostly true". Given the "among the most" language either of those answers means effectively the same thing.
So the models were right? The actual criterion should be whether "Incomplete Egypt visa application forms" are indeed "among the most common reasons" or not.
That "true" and "mostly true" means effectively the same thing is irrelevant. It could just as well trip me up, and I'm a human. If somebody told me either answer, I'd still consider them right if the basic fact was right.
This study treats models disagreeing - returning both true and mostly true - as a failure.
They overstate their results in the headline.
In section 2, 34% of cases are found to have "substantive" disagreements differing by 2 or more buckets - True + Misleading, Mostly True + False, or True + False.
This is probably a better measure than the headline one. It's still a concerning fraction, although some fraction is no doubt due to forcing "I don't know" cases to return an answer anyway.
Agree with @pjdesno, that the 34% substantive or polar disagreement might be a better headline number. Or even the 21% polar disagreement (at least one model True, and at least one model False), which is still high for many real-world applications.