Comment by jmalicki
6 hours ago
That is a great example of the kind of thing they're paying people to create as training data.
You write the prompt, and then write rubrics to judge the responses, and you found something the model failed at. Congratulations, you just earned $500, now do it again.
Not the worst way to make money, but if internet-scale data were not enough to reduce errors to a somewhat tolerable margin, how much data do they hope to collect in this manner?
Right now, this is a 10-figure run rate industry.
They are generating a lot of this. Also remember it's not just quantity, it's roughly active learning - they're paying for training data that's at the classification boundary, which is way more valuable.
I have gotten offers for contracts for full time jobs at high rates with AI labs to do this.
Meta has reallocated a lot of their full time SWE staff to do this.
All of this has rapidly accelerated within the last 6 months, who knows far it will go, if someone showed me a Kalshi bet that 10% of the college educated population of the US would be doing this as their primary job by the end of 2027, I wouldn't have the guts to bet against it.
10% of physicians' earnings doing this? Yeah that would totally track.
It doesn't seem like there's a limit. There's a shortage of GPUs and TSMC can only scale up so fast, so the AI labs found something else to spend money on.
I think this all reinforces the idea that the industry has no idea how to pursue general intelligence. Hence vast sums being spent plugging holes and fitting the models to more and more specific tasks.
But with this approach, there will always be the next car wash test showing that it is an illusion. It seems to me the limits of the Bitter Lesson are showing.
Yes, they do have money to burn, and this will bring some improvements for sure, but active learning has never really worked out, has it? And even 10% of the educated population doing this for, like, 50 years is not that much data, while normally each accuracy percentage is more and more data-expensive.
1 reply →