Comment by altairprime
6 hours ago
ELO is a bad fit because it requires competition between submitters; but if the idea is interpreted as “contributor karma score” or similar (not everyone’s familiar with the mathematical nature of ELO), then the way to close the loophole is to only consider voting inputs from the human project owner. This project chose to have people lie to a webform rather than lie to a git interface about using AI, so I don’t expect it will be particularly successful at inhibiting AI use by project-involved humans, but certainly it’ll squelch a lot of noise from unattended/passersby.
I think they were saying Elo system as kind of a general ranking system idea instead of the actual algorithm.
You could probably use some kind of pairwise ranking algorithm (like anything based on the Bradley-Terry model) to rate human vs. AI contributions, but that would take a lot of manual effort. Google is using it to (supposedly) improve their searching algorithms. They give testers two different versions and ask them what's better.