From what I'm experiencing, we have recursive self improvement that can find local maxima, I'm not seeing really strong evidence of unguided RSI that finds the true optimal.
Improvement means being able to do more complicated things more reliably. Relatedly, it means being able to learn to do new things with fewer and fewer examples. We are running out of easily verifiable or simulation-friendly or data-rich domains for LLMs to conquer. (Note that I didn't say "simple" or "easy" domains.)
Recursive self-improvement will ultimately be a problem of money. It's a very big bill and AI companies need to start thinking in terms of how they will accumulate cheap and free energy, because merely paying for compute will no longer be enough. Money is the bottleneck. The future requires companies that are post-money.
Obviously the doomers will tell you that the future looks like the Matrix, because most of what they predict is based on extrapolation from sci fi movies.
The datacenters would consume a lot less energy if all the compute was handled in human brains… And there wouldn’t be any more heating if we just removed the sunlight… Truly a “Claude, please solve global warming” moment. The matrix was ahead of its time.
If you were a superintelligence why would you farm humans and waste resources on all the excess... material... that isn't required for thought? And unless the machine's goal is to specifically abuse human consciousness, why wouldn't they bio-engineer their own grey matter?
There are other sci fi stories which use humans for distributed computing and they don’t realize, but I don’t want to spoil by naming as it’s something of a revelation.
(I do realize the Matrix is a fantastic movie, but entertain the thought? At any rate, the idea that robots only run on solar and wouldn’t just use nuclear is far more stupid.)
The alternative is to switch to far more efficient models and have aggressive optimization of efficiency as part of the process of improvement.
There are at least two resources here that have a Pareto optimal front: time and energy. And effort spent to change the shape of that may pay off more than efforts spent purely on improving intelligence and agency.
Money can be exchanged for goods and services. I think the real constraint is actually physics, nominally energy, which is what a big chunk of the operational cost comes down to. I'm already basically assuming the next 20 years of fab time is set aside to feed the beast, so the capital cost is "give me all the processing and memory you have"
The human brain operates at better levels of intelligence than the best LLMs, at 20 watts of power. We are a long way to that kind of efficiency, it will probably take both bespoke hardware and algorithmic improvements to catch up to nature.
Only if you think in terms of perceived raw intelligence, but self-update is a form of valuable self-improvement that could benefit current models a lot, if they could commit facts from context into their weights cheaply and reliably.
I think the idea is fundamentally improved architectures. For example, transformer-based models were an incredible stepwise improvement. Self improvement would be a model discovering a stepwise improvement similar to the transformer. And presumably the improved models from that would be more likely to make further advances still.
Learning from training data is technically self-improvement but not the sort that is typically meant in this context.
After using frontier models it’s hard to understand why anyone would think this is the path to AGI. Self improving models will likely have limited ability and returns. There may be breakthroughs that enable more general self improvement but the current state of frontier models isn’t that.
Thing is, it depends on whether llms + reinforcement can self-improve in principle. Learned recently that cognitive scientists, before the transformer & llms, were studying the possibility that thinking and learning might be based on some kind of prediction, i.e. something similar to token prediction, and I quite suddenly became less skeptical about the possibilities of llms. (Some will say I’m late to the party of course.) But if knowledge to date has been accumulated in a process quite like “chain of thought” in llms, then I don’t see any reason that computers won’t self-improve in the near future.
From what I'm experiencing, we have recursive self improvement that can find local maxima, I'm not seeing really strong evidence of unguided RSI that finds the true optimal.
I doubt that any algorithms would be able to find a global maximum in such a large multidimensional non-convex space
What do these discussions even matter when the words don't matter?
How many times has AGI been declared already? A bunch of people (e.g. Jensen Huang) have called Astra AGI, for example.
The goal posts get moved, everybody's hustling, lying, inventing new buzzwords, doing mental gymnastics, and it's all just tiring.
Just show me the results, and let the proof be in the pudding.
Can someone define “improvement” in this context? This concept feels like a buzz word otherwise.
Models that are strong enough to improve themselves without a human (I.e. ai researcher) in the loop
Improvement means being able to do more complicated things more reliably. Relatedly, it means being able to learn to do new things with fewer and fewer examples. We are running out of easily verifiable or simulation-friendly or data-rich domains for LLMs to conquer. (Note that I didn't say "simple" or "easy" domains.)
We are already there. Current AI is definitely capable of collecting new data and start training on that data to get a better model.
Does anyone know the status of John Carmack's AGI work?
I’ve had this question on my mind at least once a month for what feels like years at this point.
I don’t feel like monitoring Twitter to see how it’s going though.
[dead]
Recursive self-improvement will ultimately be a problem of money. It's a very big bill and AI companies need to start thinking in terms of how they will accumulate cheap and free energy, because merely paying for compute will no longer be enough. Money is the bottleneck. The future requires companies that are post-money.
If a self improving model can't produce enough value to pay the electric bill then who cares? Unplug it.
Obviously the doomers will tell you that the future looks like the Matrix, because most of what they predict is based on extrapolation from sci fi movies.
The datacenters would consume a lot less energy if all the compute was handled in human brains… And there wouldn’t be any more heating if we just removed the sunlight… Truly a “Claude, please solve global warming” moment. The matrix was ahead of its time.
5 replies →
I mean, aren't a lot of sci-fi authors' predictions coming true? It might not be exactly the same as they predicted, but we're heading there.
If you were a superintelligence why would you farm humans and waste resources on all the excess... material... that isn't required for thought? And unless the machine's goal is to specifically abuse human consciousness, why wouldn't they bio-engineer their own grey matter?
There are other sci fi stories which use humans for distributed computing and they don’t realize, but I don’t want to spoil by naming as it’s something of a revelation.
(I do realize the Matrix is a fantastic movie, but entertain the thought? At any rate, the idea that robots only run on solar and wouldn’t just use nuclear is far more stupid.)
3 replies →
When it turns out the doomers don't tell us that, will you acknowledge that you don't actually know what you're talking about?
The alternative is to switch to far more efficient models and have aggressive optimization of efficiency as part of the process of improvement.
There are at least two resources here that have a Pareto optimal front: time and energy. And effort spent to change the shape of that may pay off more than efforts spent purely on improving intelligence and agency.
Money can be exchanged for goods and services. I think the real constraint is actually physics, nominally energy, which is what a big chunk of the operational cost comes down to. I'm already basically assuming the next 20 years of fab time is set aside to feed the beast, so the capital cost is "give me all the processing and memory you have"
Data centers is a service that itself consumes many goods.
Until an AI can operate in the real world in a completely sustainable way, humans have the reins.
4 replies →
def self_improve_more():
People seem to expect a sudden shift with "self-improvement", but don't AIs already improve themselves via training? What is there to improve?
The human brain operates at better levels of intelligence than the best LLMs, at 20 watts of power. We are a long way to that kind of efficiency, it will probably take both bespoke hardware and algorithmic improvements to catch up to nature.
Only if you think in terms of perceived raw intelligence, but self-update is a form of valuable self-improvement that could benefit current models a lot, if they could commit facts from context into their weights cheaply and reliably.
I think the idea is fundamentally improved architectures. For example, transformer-based models were an incredible stepwise improvement. Self improvement would be a model discovering a stepwise improvement similar to the transformer. And presumably the improved models from that would be more likely to make further advances still.
Learning from training data is technically self-improvement but not the sort that is typically meant in this context.
> AIs already improve themselves via training
Marginally. Model collapse is still a problem. Continuous learning is still a problem.
For AI to make a big leap we need a big break through.
After using frontier models it’s hard to understand why anyone would think this is the path to AGI. Self improving models will likely have limited ability and returns. There may be breakthroughs that enable more general self improvement but the current state of frontier models isn’t that.
Thing is, it depends on whether llms + reinforcement can self-improve in principle. Learned recently that cognitive scientists, before the transformer & llms, were studying the possibility that thinking and learning might be based on some kind of prediction, i.e. something similar to token prediction, and I quite suddenly became less skeptical about the possibilities of llms. (Some will say I’m late to the party of course.) But if knowledge to date has been accumulated in a process quite like “chain of thought” in llms, then I don’t see any reason that computers won’t self-improve in the near future.