Comment by youoy
13 hours ago
By the same logic, suppose that Dario/Altman/Jensen accumulate all capital because they are the only one that have access to AGI and end up controlling democracy, turning the world into technofeudalism or whatever you want to call it. How is that better than an opensource Anthropic. At least an open source anthropic would allow to join economic forces to try to combat that situation, for example. But there are other more clear, less distopian benefits.
> By the same logic, suppose that Dario/Altman/Jensen accumulate all capital because they are the only one that have access to AGI and end up controlling democracy, turning the world into technofeudalism or whatever you want to call it. How is that better than an opensource Anthropic.
It'd be pretty bad but it's also a world where humanity lives on, which is better than a lot of other outcomes. A world where everyone has unrestricted access to AGI is like a world in which everyone has a tactical nuke in their pocket. If somehow AGI doesn't lead to x-risks, we will "merely" have to survive in a world where rogue agents can do whatever they want and defenders can only ever react. Technofeudalism would be bad, but this (technoanarchism?) is quite bad too.
However, neither does Dario seem to propose to become world dictator. Like, I'm sure he wouldn't say if he wanted it, but notice that he isn't particularly trying to aim for that outcome, either. The plan in this essay involves third-party oversight and federal control and global cooperation - IMO it's about as non-dystopian an outcome as we can hope for, if we build AGI at all.
I agree. Dario is a very smart guy and he is probably aiming for the best solution for humanity that at the same time keeps VCs/whoever gives him money happy, and keeps him in control because he does not trust the rest (probably with good reasons).
My point is more that he is aiming for a local optimum for society, but he could find a lower optimum if he was not trapped in the path that he chose early on.
I dont buy the tactical nukes/end of the world narrative. Why is he not speaking about the loss of cognitive skills of the population for example? That is a more real danger that is starting to happen. There are articles already speaking about the use of LLMs as cognitive viruses. Why are his aligment teams not reasearching and publishing about this?