Comment by atleastoptimal
2 days ago
My argument is based on
1. The first company to get AGI will likely have a multitude of high-leverage problems it would immediately put AGI to task on
2. One of those problems is simply improving itself. Another is securing that company's lead over its competitors (by basically helping every employee at that company do better at their job)
3. The company that reaches AGI for a language-style model will likely do so due to a mix of architectural tricks that can be applied to any general-purpose model, including chip design, tactical intelligence, persuasion, beating the stock market, etc
The AGI argument assumes there is a 0 -> 1 moment where the model suddenly becomes "AGI" and starts performing miraculous tasks, accompanied by recursive self-improvement. So far, our experience shows that we are getting incremental improvements over time from different companies.
These things are being commoditized, and we are still at the start of the curve when it comes to hardware, data centers, etc.
Arguing for an all-in civilization/country bet on AGI given this premise, is either foolish or a sign that you are selling "AGI"
It won't be one miraculous moment, but once a model is capable of performing tasks on its own and verifying its outputs better than a domain-expert using that AI, a human no longer being a bottleneck will allow this model to be deployed on a far larger scale, which would appear much like a 0-1 moment on the outside, even if the improvement over a previous model is minor and only involves a slight tweak of something that increases reliability.
All of that stuff takes time and resources. Self-improvement may not be easy, e.g. if they end up in a local maximum that doesn't extend, and it probably won't be cheap or fast (if it's anything like frontier LLMs it could be months of computation on enormous numbers of cutting-edge devices, costing hundreds of millions or billions, or it may not even be possible without inventing and mass-manufacturing better hardware). Another company achieving a slightly different form of AGI within a few years will probably be at least competitive, and if they have more resources or a better design they could overtake.
All the major companies are setting up to have all the hardware necessary to leverage an AGI-level system when it does emerge. The entire purpose of Starbase for OpenAI or the memphis supercomputer for X.ai is such that there would be very little delay between hitting the right marks on an AGI-capable model and deploying its capabilities en masse towards the highest-leverage aims. Certainly there will be bottlenecks, but the advantages will significantly accelerate progress.
Unless AGI includes a speed requirement, AGI is not sufficient to win the market. Take any genius in human history, the impact they had has been hugely limited by their lifespan, they didn’t solve every problem, and each discovery took them decades. The first AGIs will be the same, hyper slow for a while, giving competitors a chance to copy and stay in the race
If Einstein could replicate himself 1000x, perform all his thought experiments in parallel, not have to eat, sleep, or be limited by his human short-term memory, he would have likely have accomplished all he did far faster. AGI will at the very minimum have all the advantages of humans plus all the advantages of computers, which would be a huge automatic boon even without the fact that an AGI at human level would likely be able to scale even further.
Wait, but why would an AGI be better at improving itself compared to an AI researcher doing it?
AGI is not super intelligence.
Every frontier lab has maybe 100 top tier researchers, all of whom have limited short-term memory, need to sleep, eat, have biases, and can't rigorously parallelize themselves and run thousands of simultaneous experiments, while also modifying their own internal thought processes to a quantifiable degree. An AGI merely on the level of a mediocre AI researcher would be able to trivially leverage all these benefits, while also simply scaling its own compute (it is unlikely that the scaling gains would level out precisely at the level of a smart human)
These companies already have access to the best meat-brains in the world and what tasks do they work on? Advertisement mostly?
Ads pay the bills, but the top tier researchers are all working on top-tier AI research.