Comment by visarga

Yes, Jev is calibrated "from factory" on a bunch of tasks, but we can be almost sure our own bespoke tasks are not covered. So the model does not really know how to produce calibrated confidence scores.

What it would need is a calibration dataset on which to align. There is no calibration in the abstract, only relative to a set of test examples. A model with an uncalibrated output probability can be recalibrated using conformal prediction. You run the model over your calibration examples, get the probabilities.

Assume the new example's answer is y, and calculate its nonconformity score, higher means a worse fit. Count how many calibration examples have a score at least as high as that. Add one to this count, then divide by the total number of calibration examples plus one.