Comment by chvid
13 hours ago
"The FrontierCode verifier grades each attempt with the task's tests and rubric. We use GPT-6 Luna (medium) as the LLM judge"
Why are they using a closed source LLM as a judge? Closed source providers can see your traffic and adjust as they see fit.
Also see "Self-Preference Bias in LLM-as-a-Judge": https://arxiv.org/abs/2410.21819