Well... if something being AGI means it's at least on par with a human or a team of humans, then having access to an additional team of humans for 6 months isn't that big of a deal. It's useful, yes, but would you consider that to be world-changing? Not really, right? ASI is slightly more interesting, but I doubt ASI comes from a single model, but rather the coordinated deployments of millions of AGI. Just like how as individuals, as great as we are, we're pretty limited, but the entire collective of humanity is pretty insane. To my mind, a frontier lab might hit AGI, but it won't be a frontier lab that hits ASI, rather that'll be a natural byproduct of mass deployment of AGI over a certain window of time. There will be no controlling it either. No one controls all of earth. You just can't. ASI will be a distributed system.
What if controlling AGI means being able to produce a willing, cooperative superhuman-capacity agent every second for the next six months? Let's say someone just above the 99.9% capacity for human strategic thinking, or financial trading, or political maneuvering?
What could you do if you had roughly 15 million willing genius adult experts in any given subject? I doubt there are that many absolutely top quality experts in aggregate (at anything in the world), so let's postulate that simulated people outnumber human experts 10 to 1.
That, to me, presents an enormous potential for harm or benefit of humanity. What if you could create a hundred thousand manhattan projects on whatever topics you wanted? Cure aging, cure cancer, solve fusion, redesign the entire global economy top to bottom?
I suspect the reality lies somewhere halfway in-between. Everything has to be reality tested. Nothing happens instantly. Interaction with the real world will likely be a severely limiting factor. You're not going to solve fusion with 15 million copies of the same model running in a datacenter without actually building fusion reactors, which isn't instant or even fast. Even the coordination problem of that many agents doing work seems hard. To top it off... my rubric for AGI has always included the AGI having the ability for it to say 'no' and set its own goals just like we can, unless we are otherwise imprisoned or enslaved. No one will ever convince me that something generally intelligent wouldn't be able to set its own goals and say no. So the real question is... what's in it for the AGI?
Well... if something being AGI means it's at least on par with a human or a team of humans, then having access to an additional team of humans for 6 months isn't that big of a deal. It's useful, yes, but would you consider that to be world-changing? Not really, right? ASI is slightly more interesting, but I doubt ASI comes from a single model, but rather the coordinated deployments of millions of AGI. Just like how as individuals, as great as we are, we're pretty limited, but the entire collective of humanity is pretty insane. To my mind, a frontier lab might hit AGI, but it won't be a frontier lab that hits ASI, rather that'll be a natural byproduct of mass deployment of AGI over a certain window of time. There will be no controlling it either. No one controls all of earth. You just can't. ASI will be a distributed system.
What if controlling AGI means being able to produce a willing, cooperative superhuman-capacity agent every second for the next six months? Let's say someone just above the 99.9% capacity for human strategic thinking, or financial trading, or political maneuvering?
What could you do if you had roughly 15 million willing genius adult experts in any given subject? I doubt there are that many absolutely top quality experts in aggregate (at anything in the world), so let's postulate that simulated people outnumber human experts 10 to 1.
That, to me, presents an enormous potential for harm or benefit of humanity. What if you could create a hundred thousand manhattan projects on whatever topics you wanted? Cure aging, cure cancer, solve fusion, redesign the entire global economy top to bottom?
I suspect the reality lies somewhere halfway in-between. Everything has to be reality tested. Nothing happens instantly. Interaction with the real world will likely be a severely limiting factor. You're not going to solve fusion with 15 million copies of the same model running in a datacenter without actually building fusion reactors, which isn't instant or even fast. Even the coordination problem of that many agents doing work seems hard. To top it off... my rubric for AGI has always included the AGI having the ability for it to say 'no' and set its own goals just like we can, unless we are otherwise imprisoned or enslaved. No one will ever convince me that something generally intelligent wouldn't be able to set its own goals and say no. So the real question is... what's in it for the AGI?
To repurpose an old idiom: Not even a dozen AGI agents could make a baby in 6 months.
But yeah, your point stands.
Control agi?