Comment by aero-deck
3 years ago
Tech can start out as an extension, but what people fail to understand is that it can become slowly become autonomous over time, kinda like how a party gets out of control and people start putting holes in walls.
I think technologists rarely, if ever, talk about this because a lot of what enables our blind pursuit of technology is to say that technology must be an extension of humans and so it can never become autonomous. When ppl talk about AGI and paperclip maximizers, I think it's a way of pushing the problem far into the horizon and ignoring that the boundary is fluid and pushing.
Imagine if all the people at Google and Facebook stopped thinking that they're making the world a better place?
I think you really nailed the ultimately question: Can technology have agency?
If not, then it's just another tool for humans. We are excellent tool users, and leverage everything we can to expand our senses and abilities. We already successfully wield tools of unimaginable power.
If technology itself can have agency, then it truly is a paradigm shift for the millennia. There has never been an entity that is better at tool-use than us humans. All bets are off.
I think this is all a red herring. At least until we crack GAI, at which point paperclip maximizers and other lethal agents of pure technology come knocking.
Point being: technology, so far, has never been autonomous. But technology also doesn't grow on trees, nor does it stick out from the ground like a valuable rock. Technology is actively invented, and requires costly reproduction and maintenance. It only sticks around if enough people deem it worthy to have enough resources allocated to birth and propagate some piece of technology.
In other words: there is always someone commissioning the technology. Someone with use for it. When considering the gains and ills of progress, it is IMO wrong to focus on technology itself. Especially when talking ills, it's a good way for the actual cause of suffering to remain hidden. Every technology that ever harmed anyone was commissioned and deployed by somebody. Perhaps commissioned with ill intent from the start, or perhaps only repurposed for evil. But it's not technology that does the damage, but people - and these days, organizations, which is both government branches and businesses.
Going back to agency and autonomy - technology doesn't have agency, but people do, and importantly, large organizations seem to have separate agency of their own. Sans of GAI, no tech will turn on all humans on its own - but a corporation might, and corporations wield the most powerful of technologies.
I think Nassim Taleb used the analogy of studing an ant or a bee colony: it is not sufficient to study the ant or the bee in isolation, as it is the interactions between them and their respective colonies that shapes the behaviour. Shifting the level of analysis makes counterintuitive behaviours at the individual level (i.e. bees sacrificially stinging attackers) make sence when we shift the level of analysis up.
A corporation is just a group of humans. There's also clear governance. The CEO makes the decisions and the board of directors has oversight. The shareholders elect the board members.
It's ultimately still a group of humans making the decisions, and they are almost always rational decisions, may just not look that way from the outside with only a partial view.
1 reply →