Comment by dgellow
4 hours ago
I think you should have read the article first, at minimum the subheader
> Discounting the opinions of LLM judges with highly correlated outputs ensures that panels of judges reflect a true diversity of perspectives.
4 hours ago
I think you should have read the article first, at minimum the subheader
> Discounting the opinions of LLM judges with highly correlated outputs ensures that panels of judges reflect a true diversity of perspectives.
This doesn't even make logical sense. Actual judges have highly correlated outputs. This would literally be actively sculpting the range of opinions you want to see.
It's like the idea of political districting that thinks that the aim should be to balance each district between "the two" political parties. You're not doing anything but institutionalizing two political parties and constant conflict. You're setting the range of acceptable opinions, then choosing at random between them. Even more relevantly: when both institutionalized parties have the same opinion, it's considered the correct opinion no matter how much or how little public support it has.
> the aim should be to balance each district between "the two" political parties.
I've never heard anyone explicitly advocating for gerrymandering in favor of conflict/variance before? Is that a real thing?
The quoted sentence still leads to the same answer: no.
Because there's no "discounting of opinions". They are running a separate LLM to "score" opinions. And the result is still "no" regardless of "lineages" or "sources".
And the end of the article leads me to believe that the entire article and approach is LLM-induced garbage:
--- start quote ---
<Following a list of LLM-like suggestions>
When LLM judges agree, we should ask why. Sometimes agreement is independent evidence. Sometimes it is a shared blind spot. A good aggregation method should be able to tell the difference.
--- end quote ---