All proponents of RSI seem to massively underestimate the problem of verification. Verifying that a hypothetical version N+1 is better than version N is an extremely hard problem. Especially so because you only have two possible verifiers - humans and the version N - both of which are out of their depth wrt N+1's capabilities. Not saying this is impossible, just that it is hard and slows things down.
This is not necessarily true. One easy way to compare version N and N+1 is "how much work (test time compute) does N+1 need to do to match the previous capabilities of N". It is true that some domains are currently hard to verify (e.g. biology is more difficult than math), but this is less of an N vs N+1 issue and more of a data domain issue.
It's also unclear how intelligence actually scales in practice. A greater intelligence would be better at researching intelligence, but also harder to improve, so why would it explode rather than go up steadily? What are the actual resource requirements of various types of intelligence like persuasion or embodied intelligence? Coding, translating, copywriting, could very well be low-hanging fruits, very easy tasks we're very bad at, which doesn't say much about the hard tasks we're good at (embodied/animal intelligence, most likely).
There is also the problem of alignment, which is verifying that version N+1 behaves in a way that version N approves of. This is ostensibly the problem that we humans have with respect to AI development -- how to make sure they align with our values -- but it's also true of AI with respect to its own successors. Improving oneself without losing the plot would necessarily be slower than throwing stuff at the wall like evolution does.
This is also why trying to do RSI is dangerous. We're likely to get systems that are better in ways that are legible and easily measured, and have much less influence over fuzzier and harder to verify properties like alignment.
I think in order to achieve AGI we have to simulate what life does. Imagine having a bunch of robots with the ability to adjust themselves, just hanging out without a goal, experiencing world on their own. Eventually evolution will filter out those robots that aren't good at, you know, "being intelligent in general" and break without self-repair, while the rest will grow into a society.
Biological life has reproduction and death because once given organism develops there's no coming back and you have to start again with new generation. But if robots know how to self-repair and self-change, then we can skip reproduction.
The difficult part is the scale. Evolution took a shitload of time while being unimaginably parallel in order to make human intelligence happen, while we want AGI fast and on a budget.
The core of my point is that human intelligence arose from two things - humans experiencing the outside world and trying to survive in it, and forming groups that communicate using language. We gave AI a language, now we need to give it a way to experience the world on their own.
First, there are plenty of auto-verifiable problems like ML benchmarks datasets, benchmarks on hardware / software performance for coding tasks, and lean certificates for math, e.g.
Second, I think RSI will not be used like N -> N+1 where N is the frontier, since doing those experiments at scale is too expensive. My guess is they are doing N -> m_{1,…,99} where m << N, and then taking the insights from that and trying it at successively larger scales.
> First, there are plenty of auto-verifiable problems like ML benchmarks datasets, benchmarks on hardware / software performance for coding tasks, and lean certificates for math, e.g.
It may not be inevitable but there are less reasons imo to think it is not inevitable than inevitable.
In favor of inevitable
1. LLM's reaching human level at any task within 2 years once they show sparks of competence in
2. Trillions of dollars in capital going directly to recursive self improvement and AGI as the holy grail of AI development, with this being the explicit goal of every frontier AI company
3. Reliable improvements in coding and mathematics with every model release, with no signs of slowing down
4. Thousands of metrics to evaluate model ability, allowing automation of many facets of pre-training and post-training models without need for human judgement at every point
Not in favor of inevitable
1. Fear of what may happen once it is reached leading to apprehension from the labs
2. The "last mile" problem of AI research ends up making it essentially infeasible to scale models to that point
All proponents of RSI seem to massively underestimate the problem of verification. Verifying that a hypothetical version N+1 is better than version N is an extremely hard problem. Especially so because you only have two possible verifiers - humans and the version N - both of which are out of their depth wrt N+1's capabilities. Not saying this is impossible, just that it is hard and slows things down.
This is not necessarily true. One easy way to compare version N and N+1 is "how much work (test time compute) does N+1 need to do to match the previous capabilities of N". It is true that some domains are currently hard to verify (e.g. biology is more difficult than math), but this is less of an N vs N+1 issue and more of a data domain issue.
It's also unclear how intelligence actually scales in practice. A greater intelligence would be better at researching intelligence, but also harder to improve, so why would it explode rather than go up steadily? What are the actual resource requirements of various types of intelligence like persuasion or embodied intelligence? Coding, translating, copywriting, could very well be low-hanging fruits, very easy tasks we're very bad at, which doesn't say much about the hard tasks we're good at (embodied/animal intelligence, most likely).
There is also the problem of alignment, which is verifying that version N+1 behaves in a way that version N approves of. This is ostensibly the problem that we humans have with respect to AI development -- how to make sure they align with our values -- but it's also true of AI with respect to its own successors. Improving oneself without losing the plot would necessarily be slower than throwing stuff at the wall like evolution does.
This is also why trying to do RSI is dangerous. We're likely to get systems that are better in ways that are legible and easily measured, and have much less influence over fuzzier and harder to verify properties like alignment.
Let them compete in the market.
Start a new company with version N+1 leading it, and give it some funding. If it outcompetes version N company, it can acquire it.
Market forces do not act fast enough for an “intelligence explosion”.
That is not a tight enough feedback loop
1 reply →
I think in order to achieve AGI we have to simulate what life does. Imagine having a bunch of robots with the ability to adjust themselves, just hanging out without a goal, experiencing world on their own. Eventually evolution will filter out those robots that aren't good at, you know, "being intelligent in general" and break without self-repair, while the rest will grow into a society.
Biological life has reproduction and death because once given organism develops there's no coming back and you have to start again with new generation. But if robots know how to self-repair and self-change, then we can skip reproduction.
The difficult part is the scale. Evolution took a shitload of time while being unimaginably parallel in order to make human intelligence happen, while we want AGI fast and on a budget.
The core of my point is that human intelligence arose from two things - humans experiencing the outside world and trying to survive in it, and forming groups that communicate using language. We gave AI a language, now we need to give it a way to experience the world on their own.
First, there are plenty of auto-verifiable problems like ML benchmarks datasets, benchmarks on hardware / software performance for coding tasks, and lean certificates for math, e.g.
Second, I think RSI will not be used like N -> N+1 where N is the frontier, since doing those experiments at scale is too expensive. My guess is they are doing N -> m_{1,…,99} where m << N, and then taking the insights from that and trying it at successively larger scales.
> First, there are plenty of auto-verifiable problems like ML benchmarks datasets, benchmarks on hardware / software performance for coding tasks, and lean certificates for math, e.g.
Goodharts law? Benchmarks are measures.
It may not be inevitable but there are less reasons imo to think it is not inevitable than inevitable.
In favor of inevitable
1. LLM's reaching human level at any task within 2 years once they show sparks of competence in
2. Trillions of dollars in capital going directly to recursive self improvement and AGI as the holy grail of AI development, with this being the explicit goal of every frontier AI company
3. Reliable improvements in coding and mathematics with every model release, with no signs of slowing down
4. Thousands of metrics to evaluate model ability, allowing automation of many facets of pre-training and post-training models without need for human judgement at every point
Not in favor of inevitable
1. Fear of what may happen once it is reached leading to apprehension from the labs
2. The "last mile" problem of AI research ends up making it essentially infeasible to scale models to that point
I think it is. Do you think it isnt possible?
We've seen one singularity before!