Comment by pcwelder

There is an error in your interpretation of calibration graphs and the conclusion that Jev is more calibrated than Luna is likely incorrect. Or at least the graphs aren't enough to conclude either way.

I don't have the actual data but based on the graphs I estimate the expected calibration error to be 2-3% lower for Luna on both the tasks.

Its true that Luna seems to give lots of high confidence values but it's also correct more. Higher number of mistakes at higher confidence isn't itself undesirable if its pass rate is also high. It'd be good if they reported actual ECE.

>Most of Jev's mistakes are closer to 50% confidence, which is what you'd expect out of a well-calibrated model.

This statement is also incorrect