Comment by ajb

The real threat here is going to be when AI expands from being applied to accelerate the work of individuals, to being applied to the control of organisations. And it will be tempting to do that. We all know the limitations of managers, management hierarchies, metrics, OKRs etc. It's easy to think of a CEO deciding that all the communications between their employees should just be fed into an AI that they can query . (Ironically that would be easier to enforce if everyone was remote). It's quite possible that it would enable more effective organisations, as the CEO and upper level management can have a better idea of what is really happening. But it will reduce the already tenuous belief of the powerful that their ordinary staff are real human beings. And it will inevitably leak out from private organisations, as the executive class see no reason why they shouldn't have the same tools when running the country as when running a corporation.

Advocates of mass surveillance like to point out that no human now needs to listen to your calls. But the real danger was never the guy in a drab suit transcribing your conversions from reel-to-reel tape. It was always the chief who could call for the dossier on anyone they were finding inconvenient, and have it looked at with an eye to making you never inconvenience them again.

The full consequences of mass surveillance have not played out simply because no one had the means to process that much ad-hoc unstructured data. Now they do.