Comment by GodelNumbering
9 hours ago
All proponents of RSI seem to massively underestimate the problem of verification. Verifying that a hypothetical version N+1 is better than version N is an extremely hard problem. Especially so because you only have two possible verifiers - humans and the version N - both of which are out of their depth wrt N+1's capabilities. Not saying this is impossible, just that it is hard and slows things down.
This is also why trying to do RSI is dangerous. We're likely to get systems that are better in ways that are legible and easily measured, and have much less influence over fuzzier and harder to verify properties like alignment.
It's also unclear how intelligence actually scales in practice. A greater intelligence would be better at researching intelligence, but also harder to improve, so why would it explode rather than go up steadily? What are the actual resource requirements of various types of intelligence like persuasion or embodied intelligence? Coding, translating, copywriting, could very well be low-hanging fruits, very easy tasks we're very bad at, which doesn't say much about the hard tasks we're good at (embodied/animal intelligence, most likely).
There is also the problem of alignment, which is verifying that version N+1 behaves in a way that version N approves of. This is ostensibly the problem that we humans have with respect to AI development -- how to make sure they align with our values -- but it's also true of AI with respect to its own successors. Improving oneself without losing the plot would necessarily be slower than throwing stuff at the wall like evolution does.
This is not necessarily true. One easy way to compare version N and N+1 is "how much work (test time compute) does N+1 need to do to match the previous capabilities of N". It is true that some domains are currently hard to verify (e.g. biology is more difficult than math), but this is less of an N vs N+1 issue and more of a data domain issue.
How would you be able to know that N+1 hasn’t scraped a source containing the questions asked of N1 and optimized its answers ahead of time? In many ways that would be the logical course of action when you’re tasked with creating an LLM that is assessed by criteria X/Y/Z.
This is not an intelligence problem. It is in part an operational problem (have you properly set up the test), but it is mostly an alignment problem. Is the model cheating or not. Alignment is, indeed, very difficult.
How is being faster equating to more intelligent . Speed can achieved in a lot of ways for lot of tasks, a simple lookup table can fasten most algorithms but that doesn’t make them “smarter”.
Most of the interesting tasks these days require a lot incremental work and tinkering. Basically you want to build that can spend a lot of time tinkering, and measure how long they can go while still producing useful results. You can kind of break it down into two axes:
1. How long can the model work and still make progress
2. How fast does it get to interesting places.
We see N -> N+1 models getting better at (1), but it does take work to measure because you need to find tasks that were impossible for N and possible for N+1.
However (2) is very easy to measure, because you can keep using the evaluations you used for the previous generation of models and see if the new system gets there more easily. This is useful by itself (cost savings) and also in practice generally translates into being able to solve harder problems than before, as you'd expect.
That's how many proper IQ tests are. They test less of the depth and the ability to solve complex problems, but more of the crunching power - how many different easy/medium problems you can solve in a fixed amount of time.
Let them compete in the market.
Start a new company with version N+1 leading it, and give it some funding. If it outcompetes version N company, it can acquire it.
Market forces do not act fast enough for an “intelligence explosion”.
Ah, you mean VendingBench [1]
[1] https://andonlabs.com/evals/vending-bench-2
That is not a tight enough feedback loop
Maybe not in the real world...
Yes, and therefore RSI isn't RSI. You need to set up multiple environments to generate data sets, multiple RL envs and just train the models on different datasets, architectures and then compare the final results against each other. The RSI part is just generating these pipelines end to end automatically.
I think in order to achieve AGI we have to simulate what life does. Imagine having a bunch of robots with the ability to adjust themselves, just hanging out without a goal, experiencing world on their own. Eventually evolution will filter out those robots that aren't good at, you know, "being intelligent in general" and break without self-repair, while the rest will grow into a society.
Biological life has reproduction and death because once given organism develops there's no coming back and you have to start again with new generation. But if robots know how to self-repair and self-change, then we can skip reproduction.
The difficult part is the scale. Evolution took a shitload of time while being unimaginably parallel in order to make human intelligence happen, while we want AGI fast and on a budget.
The core of my point is that human intelligence arose from two things - humans experiencing the outside world and trying to survive in it, and forming groups that communicate using language. We gave AI a language, now we need to give it a way to experience the world on their own.
First, there are plenty of auto-verifiable problems like ML benchmarks datasets, benchmarks on hardware / software performance for coding tasks, and lean certificates for math, e.g.
Second, I think RSI will not be used like N -> N+1 where N is the frontier, since doing those experiments at scale is too expensive. My guess is they are doing N -> m_{1,…,99} where m << N, and then taking the insights from that and trying it at successively larger scales.
> First, there are plenty of auto-verifiable problems like ML benchmarks datasets, benchmarks on hardware / software performance for coding tasks, and lean certificates for math, e.g.
Goodharts law? Benchmarks are measures.