Comment by nomel
1 day ago
I see it as: they're all trying really hard problems that aren't in the training data, to gauge if their models are improving, achieved by looking at problem that humans haven't been able to solve yet. The evidence to this is all the problems being publicized in the last few weeks.
Or, less favourably: they are burning a ton of tokens on any random "hard" tasks they can think of to try and generate stories about AI doing amazing things to pump the hype, infinite monkeys theorem as a playbook. You only advertise the ones you succeed on, thus pushing the idea that's what they do routinely. Stochastic "genius".
Whether by brute force or by way of clever tricks, the fact that they reverse engineered the source of these curves is impressive.
They're building hype around their product by solving genuinely hard problems? To quote xkcd[1]: Mission. Fucking. Accomplished.
[1] https://xkcd.com/810/
The less favorable interpretation is not "genuinely hard".
If these arguably very finite amount of monkeys allow them to pull off such campaigns so frequently and consistently, these monkeys kinda do got it going on.
Finite doesn't mean small. We have no idea how much they are burning in the back to do this. If you are taking things people just don't care about enough to invest the necessary effort/money to solve, and spend way more than anyone else is willing to spend to solve them, the solution could be less AI and more access to lots of investor money you can burn on PR.
The real question is "how much would it cost me to solve a similarly hard problem?" which is impossible to answer without them giving the detail on how many of these projects they are running and how many tokens they are burning, something they don't tell us.
1 reply →
Have you noticed "infinity" in the "infinite monkey theorem"? Combined with a combinatorial explosion this means no practical usage.