Comment by tibbar
5 hours ago
This is not necessarily true. One easy way to compare version N and N+1 is "how much work (test time compute) does N+1 need to do to match the previous capabilities of N". It is true that some domains are currently hard to verify (e.g. biology is more difficult than math), but this is less of an N vs N+1 issue and more of a data domain issue.
How is being faster equating to more intelligent . Speed can achieved in a lot of ways for lot of tasks, a simple lookup table can fasten most algorithms but that doesn’t make them “smarter”.
Most of the interesting tasks these days require a lot incremental work and tinkering. Basically you want to build that can spend a lot of time tinkering, and measure how long they can go while still producing useful results. You can kind of break it down into two axes:
1. How long can the model work and still make progress
2. How fast does it get to interesting places.
We see N -> N+1 models getting better at (1), but it does take work to measure because you need to find tasks that were impossible for N and possible for N+1.
However (2) is very easy to measure, because you can keep using the evaluations you used for the previous generation of models and see if the new system gets there more easily. This is useful by itself (cost savings) and also in practice generally translates into being able to solve harder problems than before, as you'd expect.
That's how many proper IQ tests are. They test less of the depth and the ability to solve complex problems, but more of the crunching power - how many different easy/medium problems you can solve in a fixed amount of time.