Comment by tibbar

Most of the interesting tasks these days require a lot incremental work and tinkering. Basically you want to build that can spend a lot of time tinkering, and measure how long they can go while still producing useful results. You can kind of break it down into two axes:

1. How long can the model work and still make progress

2. How fast does it get to interesting places.

We see N -> N+1 models getting better at (1), but it does take work to measure because you need to find tasks that were impossible for N and possible for N+1.

However (2) is very easy to measure, because you can keep using the evaluations you used for the previous generation of models and see if the new system gets there more easily. This is useful by itself (cost savings) and also in practice generally translates into being able to solve harder problems than before, as you'd expect.