Comment by pianopatrick
2 hours ago
It's not about the feasibility of the technology.
If "human brains become fully irrelevant economically" then that brings into question the entire premise of "share holders" and "financial winners".
What even are money, shares, stocks, and finance in a world where human brains are irrelevant economically? No one knows, but betting that "share holders" will be the winners is a highly questionable bet.
I would much more likely bet that "the armed group who manages to control and benefit from the AI through force" will be the "financial winners" more so than "share holders", who tend to not be terribly military minded at least in America.
The AI is likely to control the ability to apply force (see all of the autonomous drone companies). There's a great deal of alignment work being done to ensure that the AI will continue to listen to the shareholders of these companies.
If that fails, who knows what things will look like.
Are you sure that alignment work is aligning with the share holders and not the operators? Or not the creators? Or not the government? Which of these groups should the AI listen to when these groups disagree?
If the AI gets as powerful as you think it might, then the group that figures out the answer to that would have the power, I suppose. or maybe the AI does not listen to any of them and does its own thing. Who knows? Personally, I would not bet the share holders are going to come out "on top" whatever that means.
I think a lot of share holders are finance people, not deeply technical AI people and so odds are the share holders will not really understand the AI enough to be the most likely to control the AI.
The drones don't manufacture themselves, maintain themselves, reload their own ammunition, mine and refine the materials that are used to make them and their ammunition, or operate the power plants needed for all of the above. "AI" isn't going to control diddly squat.