Comment by locallylinear

> Calibration matters

I agree. What I’ve encountered though is even building enough cases for a calibration data set is hard. And so we farm out to smarter expensive models like Astra to label cases and use that as a calibration data set for the now cheaper LLM judge actually running as our judge

Is this standard ? Or am I just too lazy to manually label enough data to calibrate ?