Comment by makeitdouble
9 days ago
Mechanical Turk exists because there is a line below which people are cheaper, even for massively parallel tasks.
If the LLM really costs less for the level of tasks that are paid for in MT right now, there sure would be a brief arbitrage period followed by the reajusting of that line I assume (of just MT shutting down if it doesn't make sense anymore)
You're forgetting completion typically isn't binary.
Take juding response pairs for DPO for example, how do you ever prove someone used ChatGPT?
ChatGPT is good enough to decide in a way that will feel internally consistent, and even if you ask MTurk users to provide their logic, ChatGPT can produce a convincing response. Eventually you're forced to start measuring noisy 2nd and 3rd order signals like "did the writing in their rationale sound like ChatGPT?"
And what's especially tough is that this affects hard to verify tasks disproportionately, while those are exactly the kinds of tasks you'd generally want MTurk for.
Yes, a very good point.
> And what's especially tough is that this affects hard to verify tasks disproportionately, while those are exactly the kinds of tasks you'd generally want MTurk for.
That's where I'd see MT just shutting down as being a very real possibility. If fraud management and consumers leaving the platform because of too much junk or unverifiable results, the whole concept could just fall apart from a business standing point.
We saw the same phenomenon I think with earlier "get paid to navigate the web" kind of scheme way back in the days, with a watch process monitoring the user actions on the computer and paying by the hour. Very quickly people found new ways to fake activity and game the system, and it all just shut down.