Comment by chatmasta
9 days ago
The funny thing is you could probably make money on Amazon Mechanical Turk by hooking it up to an LLM. We’re at this weird limbo point in history where the fraud could go either way, depending on what you think you’re paying for…
Mechanical Turk exists because there is a line below which people are cheaper, even for massively parallel tasks.
If the LLM really costs less for the level of tasks that are paid for in MT right now, there sure would be a brief arbitrage period followed by the reajusting of that line I assume (of just MT shutting down if it doesn't make sense anymore)
You're forgetting completion typically isn't binary.
Take juding response pairs for DPO for example, how do you ever prove someone used ChatGPT?
ChatGPT is good enough to decide in a way that will feel internally consistent, and even if you ask MTurk users to provide their logic, ChatGPT can produce a convincing response. Eventually you're forced to start measuring noisy 2nd and 3rd order signals like "did the writing in their rationale sound like ChatGPT?"
And what's especially tough is that this affects hard to verify tasks disproportionately, while those are exactly the kinds of tasks you'd generally want MTurk for.
Yes, a very good point.
> And what's especially tough is that this affects hard to verify tasks disproportionately, while those are exactly the kinds of tasks you'd generally want MTurk for.
That's where I'd see MT just shutting down as being a very real possibility. If fraud management and consumers leaving the platform because of too much junk or unverifiable results, the whole concept could just fall apart from a business standing point.
We saw the same phenomenon I think with earlier "get paid to navigate the web" kind of scheme way back in the days, with a watch process monitoring the user actions on the computer and paying by the hour. Very quickly people found new ways to fake activity and game the system, and it all just shut down.
I was warned and then suspended from MTurk around a decade ago while testing a workflow for audio transcription that worked a little too well. Not sure if the policies are more flexible today, but there was a lot of low hanging fruit back then.
It's pretty well known that the AI companies are heavy users of Amazon mturk for their RLHF post-training.
He meant the opposite, you could be paid to be a worker on MTurk but actually just be feeding it LLM output.
In this case it's Model Distillation. Training a new model with the outputs of a another.