Comment by femto
3 months ago
"Independent" is more important than "opposite". I don't know that promptfoo would be overtly biased. Granted they might have unconscious bias or sensitivities about offending paying customers. I do note that they present all their evidence with methods and an invitation for others to replicate or extend their results, which would go someway towards countering bias. I wouldn't trust the neutrality of someone under the influence of the CCP over promptfoo.
We’ll see soon enough, no use debating now. But I’d put money on them not showing any examples that might get them caught up in a media frenzy regarding whether they’re x-ist or anti-x-ic or anything of the sort, regardless of what the underlying ground truth in their specific questions might be.
You’ll note even on this platform, generally regarded as open and pseudo-anonymous, only a single relevant example has been put forward.